Selective usage of inference models based on visual content

ABSTRACT

Systems, methods and non-transitory computer readable media for image processing are provided. Images may be obtained, for example by capturing the images using an image sensor. The images may be analyzed to identify scene information. An inference model may be obtained based on the scene information. The scene information may be used to determine parameters for modifying the obtained inference model. Additional images may be obtained, for example using the image sensor. The additional images may be analyzed using the inference model.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/782,814, filed Feb. 5, 2020, which is a continuation of U.S. patentapplication Ser. No. 15/864,006, filed on Jan. 8, 2018, which claims thebenefit of priority of U.S. Provisional Patent Application No.62/444,001, filed on Jan. 9, 2017, and U.S. Provisional PatentApplication No. 62/452,707, filed on Jan. 31, 2017.

The entire contents of all of the above-identified applications areherein incorporated by reference.

BACKGROUND Technological Field

The disclosed embodiments generally relate to systems and methods forimage processing. More particularly, the disclosed embodiments relate tosystems and methods for selective image processing based on type ofvisual content.

Background Information

Image sensors are now part of numerous devices, from security systems tomobile phones, and the availability of images and videos produced bythose devices is increasing.

SUMMARY

In some embodiments, systems and methods for image processing areprovided.

In some embodiments, a first and a second group of images may beobtained, for example by capturing the images using an image sensor; thefirst group of images may be analyzed to identify objects in theenvironment; a first and a second regions of the second group of imagesmay be identified based on the identified objects; a processing schememay be selected based on the identified objects; the first region may beprocessed using the selected processing scheme, and the second regionmay be processed using a different processing scheme.

In some embodiments, a first and a second group of images may beobtained, for example by capturing the images using an image sensor; thefirst group of images may be analyzed to obtain scene information; aninference model may be selected based on the scene information; and thesecond group of images may be processed using the selected inferencemodel.

In some embodiments, a stream of images may be obtained, for example bycapturing images using an image sensor; points in time associated withan activity may be obtained; for each point in time, the stream ofimages may be analyzed to identify events related to the activity andpreceding the point in time; and based on the identified events, anevent detection rule configured to analyze images to detect at least oneevent may be obtained.

In some embodiments, image based information may be obtained; the imagebased information may be analyzed to identify instances of a repeatedactivity of a selected person; and properties of the repeated activityof the selected person may be determined based on the identifiedinstances of the repeated activity.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are block diagrams illustrating some possibleimplementations of a communicating system.

FIGS. 2A and 2B are block diagrams illustrating some possibleimplementations of an apparatus.

FIG. 3 is a block diagram illustrating a possible implementation of aserver.

FIGS. 4A and 4B are block diagrams illustrating some possibleimplementations of a cloud platform.

FIG. 5 is a block diagram illustrating a possible implementation of acomputational node.

FIG. 6 illustrates an example of a process for selective imageprocessing.

FIG. 7 illustrates an example of a process for selective use ofinference models.

FIG. 8A is a schematic illustration of an example of an environment of aroom.

FIG. 8B is a schematic illustration of an example of an environment of ayard.

FIG. 9 illustrates an example of a process for facilitating learning ofvisual events.

FIGS. 10A, 10B, 10C and 10D are schematic illustrations of exampleimages captured by an apparatus consistent with an embodiment of thepresent disclosure.

FIG. 11 illustrates an example of a process for collecting informationabout repeated behavior.

FIGS. 12A, 12B, 12C, 12D, 12E and 12F are schematic illustrations ofexample images captured by an apparatus consistent with an embodiment ofthe present disclosure.

DESCRIPTION

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the specificationdiscussions utilizing terms such as “processing”, “calculating”,“computing”, “determining”, “generating”, “setting”, “configuring”,“selecting”, “defining”, “applying”, “obtaining”, “monitoring”,“providing”, “identifying”, “segmenting”, “classifying”, “analyzing”,“associating”, “extracting”, “storing”, “receiving”, “transmitting”, orthe like, include action and/or processes of a computer that manipulateand/or transform data into other data, said data represented as physicalquantities, for example such as electronic quantities, and/or said datarepresenting the physical objects. The terms “computer”, “processor”,“controller”, “processing unit”, “computing unit”, and “processingmodule” should be expansively construed to cover any kind of electronicdevice, component or unit with data processing capabilities, including,by way of non-limiting example, a personal computer, a wearablecomputer, a tablet, a smartphone, a server, a computing system, a cloudcomputing platform, a communication device, a processor (for example,digital signal processor (DSP), an image signal processor (ISR), amicrocontroller, a field programmable gate array (FPGA), an applicationspecific integrated circuit (ASIC), a central processing unit (CPA), agraphics processing unit (GPU), a visual processing unit (VPU), and soon), possibly with embedded memory, a single core processor, a multicore processor, a core within a processor, any other electroniccomputing device, or any combination of the above.

The operations in accordance with the teachings herein may be performedby a computer specially constructed or programmed to perform thedescribed functions.

As used herein, the phrase “for example,” “such as”, “for instance” andvariants thereof describe non-limiting embodiments of the presentlydisclosed subject matter. Reference in the specification to “one case”,“some cases”, “other cases” or variants thereof means that a particularfeature, structure or characteristic described in connection with theembodiment(s) may be included in at least one embodiment of thepresently disclosed subject matter. Thus the appearance of the phrase“one case”, “some cases”, “other cases” or variants thereof does notnecessarily refer to the same embodiment(s). As used herein, the term“and/or” includes any and all combinations of one or more of theassociated listed items.

It is appreciated that certain features of the presently disclosedsubject matter, which are, for clarity, described in the context ofseparate embodiments, may also be provided in combination in a singleembodiment. Conversely, various features of the presently disclosedsubject matter, which are, for brevity, described in the context of asingle embodiment, may also be provided separately or in any suitablesub-combination.

The term “image sensor” is recognized by those skilled in the art andrefers to any device configured to capture images, a sequence of images,videos, and so forth. This includes sensors that convert optical inputinto images, where optical input can be visible light (like in acamera), radio waves, microwaves, terahertz waves, ultraviolet light,infrared light, x-rays, gamma rays, and/or any other light spectrum.This also includes both 2D and 3D sensors. Examples of image sensortechnologies may include: CCD, CMOS, NMOS, and so forth. 3D sensors maybe implemented using different technologies, including: stereo camera,active stereo camera, time of flight camera, structured light camera,radar, range image camera, and so forth.

The term “audio sensor” is recognized by those skilled in the art andrefers to any device configured to capture audio data. This includessensors that convert audio and sounds into digital audio data.

The term “electrical impedance sensor” is recognized by those skilled inthe art and refers to any sensor configured to measure the electricalconnectivity and/or permittivity between two or more points. Thisinclude but not limited to: sensors configured to measuring changes inconnectivity and/or permittivity over time; sensors configured tomeasure the connectivity and/or permittivity of biological tissues;sensors configured to measure the connectivity and/or permittivity ofparts of body based, at least in part, on the connectivity and/orpermittivity between surface electrodes; sensors configured to provideElectrical Impedance Tomography images, and so forth. Such sensors mayinclude but not limited to: sensors that apply alternating currents at asingle frequency; sensors that apply alternating currents at multiplefrequencies; and so forth. Additionally, this may also include sensorsthat measure the electrical resistance between two or more points, whichare sometimes referred to as ohmmeter.

In embodiments of the presently disclosed subject matter, one or morestages illustrated in the figures may be executed in a different orderand/or one or more groups of stages may be executed simultaneously andvice versa. The figures illustrate a general schematic of the systemarchitecture in accordance embodiments of the presently disclosedsubject matter. Each module in the figures can be made up of anycombination of software, hardware and/or firmware that performs thefunctions as defined and explained herein. The modules in the figuresmay be centralized in one location or dispersed over more than onelocation.

It should be noted that some examples of the presently disclosed subjectmatter are not limited in application to the details of construction andthe arrangement of the components set forth in the following descriptionor illustrated in the drawings. The invention can be capable of otherembodiments or of being practiced or carried out in various ways. Also,it is to be understood that the phraseology and terminology employedherein is for the purpose of description and should not be regarded aslimiting.

In this document, an element of a drawing that is not described withinthe scope of the drawing and is labeled with a numeral that has beendescribed in a previous drawing may have the same use and description asin the previous drawings.

The drawings in this document may not be to any scale. Different figuresmay use different scales and different scales can be used even withinthe same drawing, for example different scales for different views ofthe same object or different scales for the two adjacent objects.

FIG. 1A is a block diagram illustrating a possible implementation of acommunicating system. In this example, apparatuses 200 a and 200 b maycommunicate with server 300 a, with server 300 b, with cloud platform400, with each other, and so forth. Possible implementations ofapparatuses 200 a and 200 b may include apparatus 200 as described inFIGS. 2A and 2B. Possible implementations of servers 300 a and 300 b mayinclude server 300 as described in FIG. 3. Some possible implementationsof cloud platform 400 are described in FIGS. 4A, 4B and 5. In thisexample apparatuses 200 a and 200 b may communicate directly with mobilephone 111, tablet 112, and personal computer (PC) 113. Apparatuses 200 aand 200 b may communicate with local router 120 directly, and/or throughat least one of mobile phone 111, tablet 112, and personal computer (PC)113. In this example, local router 120 may be connected with acommunication network 130. Examples of communication network 130 mayinclude the Internet, phone networks, cellular networks, satellitecommunication networks, private communication networks, virtual privatenetworks (VPN), and so forth. Apparatuses 200 a and 200 b may connect tocommunication network 130 through local router 120 and/or directly.Apparatuses 200 a and 200 b may communicate with other devices, such asservers 300 a, server 300 b, cloud platform 400, remote storage 140 andnetwork attached storage (NAS) 150, through communication network 130and/or directly.

FIG. 1B is a block diagram illustrating a possible implementation of acommunicating system. In this example, apparatuses 200 a, 200 b and 200c may communicate with cloud platform 400 and/or with each other throughcommunication network 130. Possible implementations of apparatuses 200a, 200 b and 200 c may include apparatus 200 as described in FIGS. 2Aand 2B. Some possible implementations of cloud platform 400 aredescribed in FIGS. 4A, 4B and 5.

FIGS. 1A and 1B illustrate some possible implementations of acommunication system. In some embodiments, other communication systemsthat enable communication between apparatus 200 and server 300 may beused. In some embodiments, other communication systems that enablecommunication between apparatus 200 and cloud platform 400 may be used.In some embodiments, other communication systems that enablecommunication among a plurality of apparatuses 200 may be used.

FIG. 2A is a block diagram illustrating a possible implementation ofapparatus 200. In this example, apparatus 200 may comprise: one or morememory units 210, one or more processing units 220, and one or morecommunication modules 230. In some implementations, apparatus 200 maycomprise additional components, while some components listed above maybe excluded.

FIG. 2B is a block diagram illustrating a possible implementation ofapparatus 200. In this example, apparatus 200 may comprise: one or morememory units 210, one or more processing units 220, one or morecommunication modules 230, one or more power sources 240, one or moreaudio sensors 250, one or more image sensors 260, one or more lightsources 265, one or more motion sensors 270, and one or more positioningsensors 275. In some implementations, apparatus 200 may compriseadditional components, while some components listed above may beexcluded. For example, in some implementations apparatus 200 may alsocomprise at least one of the following: one or more barometers; one ormore pressure sensors; one or more proximity sensors; one or moreelectrical impedance sensors; one or more electrical voltage sensors;one or more electrical current sensors; one or more user input devices;one or more output devices; and so forth. In another example, in someimplementations at least one of the following may be excluded fromapparatus 200: memory units 210, communication modules 230, powersources 240, audio sensors 250, image sensors 260, light sources 265,motion sensors 270, and positioning sensors 275.

In some embodiments, one or more power sources 240 may be configured to:power apparatus 200; power server 300; power cloud platform 400; and/orpower computational node 500. Possible implementation examples of powersources 240 may include: one or more electric batteries; one or morecapacitors; one or more connections to external power sources; one ormore power convertors; any combination of the above; and so forth.

In some embodiments, the one or more processing units 220 may beconfigured to execute software programs. For example, processing units220 may be configured to execute software programs stored on the memoryunits 210. In some cases, the executed software programs may storeinformation in memory units 210. In some cases, the executed softwareprograms may retrieve information from the memory units 210. Possibleimplementation examples of the processing units 220 may include: one ormore single core processors, one or more multicore processors; one ormore controllers; one or more application processors; one or more systemon a chip processors; one or more central processing units; one or moregraphical processing units; one or more neural processing units; anycombination of the above; and so forth.

In some embodiments, the one or more communication modules 230 may beconfigured to receive and transmit information. For example, controlsignals may be transmitted and/or received through communication modules230. In another example, information received though communicationmodules 230 may be stored in memory units 210. In an additional example,information retrieved from memory units 210 may be transmitted usingcommunication modules 230. In another example, input data may betransmitted and/or received using communication modules 230. Examples ofsuch input data may include: input data inputted by a user using userinput devices; information captured using one or more sensors; and soforth. Examples of such sensors may include: audio sensors 250; imagesensors 260; motion sensors 270; positioning sensors 275; chemicalsensors; temperature sensors; barometers; pressure sensors; proximitysensors; electrical impedance sensors; electrical voltage sensors;electrical current sensors; and so forth.

In some embodiments, the one or more audio sensors 250 may be configuredto capture audio by converting sounds to digital information. Someexamples of audio sensors 250 may include: microphones, unidirectionalmicrophones, bidirectional microphones, cardioid microphones,omnidirectional microphones, onboard microphones, wired microphones,wireless microphones, any combination of the above, and so forth. Insome examples, the captured audio may be stored in memory units 210. Insome additional examples, the captured audio may be transmitted usingcommunication modules 230, for example to other computerized devices,such as server 300, cloud platform 400, computational node 500, and soforth. In some examples, processing units 220 may control the aboveprocesses. For example, processing units 220 may control at least oneof: capturing of the audio; storing the captured audio; transmitting ofthe captured audio; and so forth. In some cases, the captured audio maybe processed by processing units 220. For example, the captured audiomay be compressed by processing units 220; possibly followed: by storingthe compressed captured audio in memory units 210; by transmitted thecompressed captured audio using communication modules 230; and so forth.In another example, the captured audio may be processed using speechrecognition algorithms. In another example, the captured audio may beprocessed using speaker recognition algorithms.

In some embodiments, the one or more image sensors 260 may be configuredto capture visual information by converting light to: images; sequenceof images; videos; and so forth. In some examples, the captured visualinformation may be stored in memory units 210. In some additionalexamples, the captured visual information may be transmitted usingcommunication modules 230, for example to other computerized devices,such as server 300, cloud platform 400, computational node 500, and soforth. In some examples, processing units 220 may control the aboveprocesses. For example, processing units 220 may control at least oneof: capturing of the visual information; storing the captured visualinformation; transmitting of the captured visual information; and soforth. In some cases, the captured visual information may be processedby processing units 220. For example, the captured visual informationmay be compressed by processing units 220; possibly followed: by storingthe compressed captured visual information in memory units 210; bytransmitted the compressed captured visual information usingcommunication modules 230; and so forth. In another example, thecaptured visual information may be processed in order to: detectobjects, detect events, detect action, detect face, detect people,recognize person, and so forth.

In some embodiments, the one or more light sources 265 may be configuredto emit light, for example in order to enable better image capturing byimage sensors 260. In some examples, the emission of light may becoordinated with the capturing operation of image sensors 260. In someexamples, the emission of light may be continuous. In some examples, theemission of light may be performed at selected times. The emitted lightmay be visible light, infrared light, x-rays, gamma rays, and/or in anyother light spectrum.

In some embodiments, the one or more motion sensors 270 may beconfigured to perform at least one of the following: detect motion ofobjects in the environment of apparatus 200; measure the velocity ofobjects in the environment of apparatus 200; measure the acceleration ofobjects in the environment of apparatus 200; detect motion of apparatus200; measure the velocity of apparatus 200; measure the acceleration ofapparatus 200; and so forth. In some implementations, the one or moremotion sensors 270 may comprise one or more accelerometers configured todetect changes in proper acceleration and/or to measure properacceleration of apparatus 200. In some implementations, the one or moremotion sensors 270 may comprise one or more gyroscopes configured todetect changes in the orientation of apparatus 200 and/or to measureinformation related to the orientation of apparatus 200. In someimplementations, motion sensors 270 may be implemented using imagesensors 260, for example by analyzing images captured by image sensors260 to perform at least one of the following tasks: track objects in theenvironment of apparatus 200; detect moving objects in the environmentof apparatus 200; measure the velocity of objects in the environment ofapparatus 200; measure the acceleration of objects in the environment ofapparatus 200; measure the velocity of apparatus 200, for example bycalculating the egomotion of image sensors 260; measure the accelerationof apparatus 200, for example by calculating the egomotion of imagesensors 260; and so forth. In some implementations, motion sensors 270may be implemented using image sensors 260 and light sources 265, forexample by implementing a LIDAR using image sensors 260 and lightsources 265. In some implementations, motion sensors 270 may beimplemented using one or more RADARs. In some examples, informationcaptured using motion sensors 270: may be stored in memory units 210,may be processed by processing units 220, may be transmitted and/orreceived using communication modules 230, and so forth.

In some embodiments, the one or more positioning sensors 275 may beconfigured to obtain positioning information of apparatus 200, to detectchanges in the position of apparatus 200, and/or to measure the positionof apparatus 200. In some examples, positioning sensors 275 may beimplemented using one of the following technologies: Global PositioningSystem (GPS), GLObal NAvigation Satellite System (GLONASS), Galileoglobal navigation system, BeiDou navigation system, other GlobalNavigation Satellite Systems (GNSS), Indian Regional NavigationSatellite System (IRNSS), Local Positioning Systems (LPS), Real-TimeLocation Systems (RTLS), Indoor Positioning System (IPS), Wi-Fi basedpositioning systems, cellular triangulation, and so forth. In someexamples, information captured using positioning sensors 275 may bestored in memory units 210, may be processed by processing units 220,may be transmitted and/or received using communication modules 230, andso forth.

In some embodiments, the one or more chemical sensors may be configuredto perform at least one of the following: measure chemical properties inthe environment of apparatus 200; measure changes in the chemicalproperties in the environment of apparatus 200; detect the present ofchemicals in the environment of apparatus 200; measure the concentrationof chemicals in the environment of apparatus 200. Examples of suchchemical properties may include: pH level, toxicity, temperature, and soforth. Examples of such chemicals may include: electrolytes, particularenzymes, particular hormones, particular proteins, smoke, carbondioxide, carbon monoxide, oxygen, ozone, hydrogen, hydrogen sulfide, andso forth. In some examples, information captured using chemical sensorsmay be stored in memory units 210, may be processed by processing units220, may be transmitted and/or received using communication modules 230,and so forth.

In some embodiments, the one or more temperature sensors may beconfigured to detect changes in the temperature of the environment ofapparatus 200 and/or to measure the temperature of the environment ofapparatus 200. In some examples, information captured using temperaturesensors may be stored in memory units 210, may be processed byprocessing units 220, may be transmitted and/or received usingcommunication modules 230, and so forth.

In some embodiments, the one or more barometers may be configured todetect changes in the atmospheric pressure in the environment ofapparatus 200 and/or to measure the atmospheric pressure in theenvironment of apparatus 200. In some examples, information capturedusing the barometers may be stored in memory units 210, may be processedby processing units 220, may be transmitted and/or received usingcommunication modules 230, and so forth.

In some embodiments, the one or more pressure sensors may be configuredto perform at least one of the following: detect pressure in theenvironment of apparatus 200; measure pressure in the environment ofapparatus 200; detect change in the pressure in the environment ofapparatus 200; measure change in pressure in the environment ofapparatus 200; detect pressure at a specific point and/or region of thesurface area of apparatus 200; measure pressure at a specific pointand/or region of the surface area of apparatus 200; detect change inpressure at a specific point and/or area; measure change in pressure ata specific point and/or region of the surface area of apparatus 200;measure the pressure differences between two specific points and/orregions of the surface area of apparatus 200; measure changes inrelative pressure between two specific points and/or regions of thesurface area of apparatus 200. In some examples, information capturedusing the pressure sensors may be stored in memory units 210, may beprocessed by processing units 220, may be transmitted and/or receivedusing communication modules 230, and so forth.

In some embodiments, the one or more proximity sensors may be configuredto perform at least one of the following: detect contact of a solidobject with the surface of apparatus 200; detect contact of a solidobject with a specific point and/or region of the surface area ofapparatus 200; detect a proximity of apparatus 200 to an object. In someimplementations, proximity sensors may be implemented using imagesensors 260 and light sources 265, for example by emitting light usinglight sources 265, such as ultraviolet light, visible light, infraredlight and/or microwave light, and detecting the light reflected fromnearby objects using image sensors 260 to detect the present of nearbyobjects. In some examples, information captured using the proximitysensors may be stored in memory units 210, may be processed byprocessing units 220, may be transmitted and/or received usingcommunication modules 230, and so forth.

In some embodiments, the one or more electrical impedance sensors may beconfigured to perform at least one of the following: detect change overtime in the connectivity and/or permittivity between two electrodes;measure changes over time in the connectivity and/or permittivitybetween two electrodes; capture Electrical Impedance Tomography (EIT)images. In some examples, information captured using the electricalimpedance sensors may be stored in memory units 210, may be processed byprocessing units 220, may be transmitted and/or received usingcommunication modules 230, and so forth.

In some embodiments, the one or more electrical voltage sensors may beconfigured to perform at least one of the following: detect and/ormeasure voltage between two electrodes; detect and/or measure changesover time in the voltage between two electrodes. In some examples,information captured using the electrical voltage sensors may be storedin memory units 210, may be processed by processing units 220, may betransmitted and/or received using communication modules 230, and soforth.

In some embodiments, the one or more electrical current sensors may beconfigured to perform at least one of the following: detect and/ormeasure electrical current flowing between two electrodes; detect and/ormeasure changes over time in the electrical current flowing between twoelectrodes. In some examples, information captured using the electricalcurrent sensors may be stored in memory units 210, may be processed byprocessing units 220, may be transmitted and/or received usingcommunication modules 230, and so forth.

In some embodiments, the one or more user input devices may beconfigured to allow one or more users to input information. In someexamples, user input devices may comprise at least one of the following:a keyboard, a mouse, a touch pad, a touch screen, a joystick, amicrophone, an image sensor, and so forth. In some examples, the userinput may be in the form of at least one of: text, sounds, speech, handgestures, body gestures, tactile information, and so forth. In someexamples, the user input may be stored in memory units 210, may beprocessed by processing units 220, may be transmitted and/or receivedusing communication modules 230, and so forth.

In some embodiments, the one or more user output devices may beconfigured to provide output information to one or more users. In someexamples, such output information may comprise of at least one of:notifications, feedbacks, reports, and so forth. In some examples, useroutput devices may comprise at least one of: one or more audio outputdevices; one or more textual output devices; one or more visual outputdevices; one or more tactile output devices; and so forth. In someexamples, the one or more audio output devices may be configured tooutput audio to a user, for example through: a headset, a set ofspeakers, and so forth. In some examples, the one or more visual outputdevices may be configured to output visual information to a user, forexample through: a display screen, an augmented reality display system,a printer, a LED indicator, and so forth. In some examples, the one ormore tactile output devices may be configured to output tactilefeedbacks to a user, for example through vibrations, through motions, byapplying forces, and so forth. In some examples, the output may beprovided: in real time, offline, automatically, upon request, and soforth. In some examples, the output information may be read from memoryunits 210, may be provided by a software executed by processing units220, may be transmitted and/or received using communication modules 230,and so forth.

FIG. 3 is a block diagram illustrating a possible implementation ofserver 300. In this example, server 300 may comprise: one or more memoryunits 210, one or more processing units 220, one or more communicationmodules 230, and one or more power sources 240. In some implementations,server 300 may comprise additional components, while some componentslisted above may be excluded. For example, in some implementationsserver 300 may also comprise at least one of the following: one or moreuser input devices; one or more output devices; and so forth. In anotherexample, in some implementations at least one of the following may beexcluded from server 300: memory units 210, communication modules 230,and power sources 240.

FIG. 4A is a block diagram illustrating a possible implementation ofcloud platform 400. In this example, cloud platform 400 may comprisecomputational node 500 a, computational node 500 b, computational node500 c and computational node 500 d. In some examples, a possibleimplementation of computational nodes 500 a, 500 b, 500 c and 500 d maycomprise server 300 as described in FIG. 3. In some examples, a possibleimplementation of computational nodes 500 a, 500 b, 500 c and 500 d maycomprise computational node 500 as described in FIG. 5.

FIG. 4B is a block diagram illustrating a possible implementation ofcloud platform 400. In this example, cloud platform 400 may comprise:one or more computational nodes 500, one or more shared memory modules410, one or more power sources 240, one or more node registrationmodules 420, one or more load balancing modules 430, one or moreinternal communication modules 440, and one or more externalcommunication modules 450. In some implementations, cloud platform 400may comprise additional components, while some components listed abovemay be excluded. For example, in some implementations cloud platform 400may also comprise at least one of the following: one or more user inputdevices; one or more output devices; and so forth. In another example,in some implementations at least one of the following may be excludedfrom cloud platform 400: shared memory modules 410, power sources 240,node registration modules 420, load balancing modules 430, internalcommunication modules 440, and external communication modules 450.

FIG. 5 is a block diagram illustrating a possible implementation ofcomputational node 500. In this example, computational node 500 maycomprise: one or more memory units 210, one or more processing units220, one or more shared memory access modules 510, one or more powersources 240, one or more internal communication modules 440, and one ormore external communication modules 450. In some implementations,computational node 500 may comprise additional components, while somecomponents listed above may be excluded. For example, in someimplementations computational node 500 may also comprise at least one ofthe following: one or more user input devices; one or more outputdevices; and so forth. In another example, in some implementations atleast one of the following may be excluded from computational node 500:memory units 210, shared memory access modules 510, power sources 240,internal communication modules 440, and external communication modules450.

In some embodiments, internal communication modules 440 and externalcommunication modules 450 may be implemented as a combined communicationmodule, such as communication modules 230. In some embodiments, onepossible implementation of cloud platform 400 may comprise server 300.In some embodiments, one possible implementation of computational node500 may comprise server 300. In some embodiments, one possibleimplementation of shared memory access modules 510 may comprise usinginternal communication modules 440 to send information to shared memorymodules 410 and/or receive information from shared memory modules 410.In some embodiments, node registration modules 420 and load balancingmodules 430 may be implemented as a combined module.

In some embodiments, the one or more shared memory modules 410 may beaccessed by more than one computational node. Therefore, shared memorymodules 410 may allow information sharing among two or morecomputational nodes 500. In some embodiments, the one or more sharedmemory access modules 510 may be configured to enable access ofcomputational nodes 500 and/or the one or more processing units 220 ofcomputational nodes 500 to shared memory modules 410. In some examples,computational nodes 500 and/or the one or more processing units 220 ofcomputational nodes 500, may access shared memory modules 410, forexample using shared memory access modules 510, in order to perform atleast one of: executing software programs stored on shared memorymodules 410, store information in shared memory modules 410, retrieveinformation from the shared memory modules 410.

In some embodiments, the one or more node registration modules 420 maybe configured to track the availability of the computational nodes 500.In some examples, node registration modules 420 may be implemented as: asoftware program, such as a software program executed by one or more ofthe computational nodes 500; a hardware solution; a combined softwareand hardware solution; and so forth. In some implementations, noderegistration modules 420 may communicate with computational nodes 500,for example using internal communication modules 440. In some examples,computational nodes 500 may notify node registration modules 420 oftheir status, for example by sending messages: at computational node 500startup; at computational node 500 shutdown; at constant intervals; atselected times; in response to queries received from node registrationmodules 420; and so forth. In some examples, node registration modules420 may query about computational nodes 500 status, for example bysending messages: at node registration module 420 startup; at constantintervals; at selected times; and so forth.

In some embodiments, the one or more load balancing modules 430 may beconfigured to divide the work load among computational nodes 500. Insome examples, load balancing modules 430 may be implemented as: asoftware program, such as a software program executed by one or more ofthe computational nodes 500; a hardware solution; a combined softwareand hardware solution; and so forth. In some implementations, loadbalancing modules 430 may interact with node registration modules 420 inorder to obtain information regarding the availability of thecomputational nodes 500. In some implementations, load balancing modules430 may communicate with computational nodes 500, for example usinginternal communication modules 440. In some examples, computationalnodes 500 may notify load balancing modules 430 of their status, forexample by sending messages: at computational node 500 startup; atcomputational node 500 shutdown; at constant intervals; at selectedtimes; in response to queries received from load balancing modules 430;and so forth. In some examples, load balancing modules 430 may queryabout computational nodes 500 status, for example by sending messages:at load balancing module 430 startup; at constant intervals; at selectedtimes; and so forth.

In some embodiments, the one or more internal communication modules 440may be configured to receive information from one or more components ofcloud platform 400, and/or to transmit information to one or morecomponents of cloud platform 400. For example, control signals and/orsynchronization signals may be sent and/or received through internalcommunication modules 440. In another example, input information forcomputer programs, output information of computer programs, and/orintermediate information of computer programs, may be sent and/orreceived through internal communication modules 440. In another example,information received though internal communication modules 440 may bestored in memory units 210, in shared memory units 410, and so forth. Inan additional example, information retrieved from memory units 210and/or shared memory units 410 may be transmitted using internalcommunication modules 440. In another example, input data may betransmitted and/or received using internal communication modules 440.Examples of such input data may include input data inputted by a userusing user input devices.

In some embodiments, the one or more external communication modules 450may be configured to receive and/or to transmit information. Forexample, control signals may be sent and/or received through externalcommunication modules 450. In another example, information receivedthough external communication modules 450 may be stored in memory units210, in shared memory units 410, and so forth. In an additional example,information retrieved from memory units 210 and/or shared memory units410 may be transmitted using external communication modules 450. Inanother example, input data may be transmitted and/or received usingexternal communication modules 450. Examples of such input data mayinclude: input data inputted by a user using user input devices;information captured from the environment of apparatus 200 using one ormore sensors; and so forth. Examples of such sensors may include: audiosensors 250; image sensors 260; motion sensors 270; positioning sensors275; chemical sensors; temperature sensors; barometers; pressuresensors; proximity sensors; electrical impedance sensors; electricalvoltage sensors; electrical current sensors; and so forth.

FIG. 6 illustrates an example of process 600 for selective imageprocessing. In some examples, process 600, as well as all individualsteps therein, may be performed by various aspects of: apparatus 200;server 300; cloud platform 400; computational node 500; and so forth.For example, process 600 may be performed by processing units 220,executing software instructions stored within memory units 210 and/orwithin shared memory modules 410. In this example, process 600 maycomprise: obtaining first group of images (Step 610); identifyingobjects in the first group of images (Step 620); obtaining second groupof images (Step 630); identifying regions in the second group of images(Step 640); selecting processing schemes (Step 650); and processing theidentified regions (Step 660). In some implementations, process 600 maycomprise one or more additional steps, while some of the steps listedabove may be modified or excluded. For example, in some cases Step 650may be excluded from process 600. In some implementations, one or moresteps illustrated in FIG. 6 may be executed in a different order and/orone or more groups of steps may be executed simultaneously and viceversa. For example, Step 630 and/or Step 640 may be executed before,after and/or simultaneously with Step 610 and/or Step 620; Step 650 maybe executed before, after and/or simultaneously with Step 610 and/orStep 620 and/or Step 630 and/or Step 640; Step 660 may be executed afterand/or simultaneously with Step 640 and/or Step 650, and so forth.Examples of possible execution manners of process 600 may include:continuous execution, returning to the beginning of the process once theprocess normal execution ends; periodically execution, executing theprocess at selected times; execution upon the detection of a trigger,where examples of such trigger may include trigger from a user, triggerfrom another process, trigger from an external device, etc.; anycombination of the above; and so forth.

In some embodiments, obtaining first group of images (Step 610) and/orobtaining second group of images (Step 630) and/or obtaining a stream ofimages (Step 910) and/or receiving image data (Step 1110) may compriseobtaining image data captured using image sensors (such as image sensors260). Some examples of such image data may include: images; segments ofimages; sequence of images; video clips; segments of video clips; videostreams; segments of video streams; information based, at least in part,on any of the above; any combination of the above; and so forth.

In some examples, Step 610 and/or Step 630 and/or Step 910 and/or Step1110 may comprise, in addition or alternatively to obtaining image dataand/or other input data, obtaining audio data captured using audiosensors (such as audio sensors 250). Examples of audio data may include:audio recordings; segments of audio recordings; audio streams; segmentsof audio streams; information based, at least in part, on any of theabove; any combination of the above; and so forth.

In some examples, Step 610 and/or Step 630 and/or Step 910 and/or Step1110 may comprise, in addition or alternatively to obtaining image dataand/or other input data, obtaining motion information captured usingmotion sensors (such as motion sensors 270). Examples of such motioninformation may include: indications related to motion of objects;measurements related to the velocity of objects; measurements related tothe acceleration of objects; indications related to motion of motionsensor 270; measurements related to the velocity of motion sensor 270;measurements related to the acceleration of motion sensor 270;information based, at least in part, on any of the above; anycombination of the above; and so forth.

In some examples, Step 610 and/or Step 630 and/or Step 910 and/or Step1110 may comprise, in addition or alternatively to obtaining image dataand/or other input data, obtaining position information captured usingpositioning sensors (such as positioning sensors 275). Examples of suchposition information may include: indications related to the position ofpositioning sensors 275; indications related to changes in the positionof positioning sensors 275; measurements related to the position ofpositioning sensors 275; indications related to the orientation ofpositioning sensors 275; indications related to changes in theorientation of positioning sensors 275; measurements related to theorientation of positioning sensors 275; measurements related to changesin the orientation of positioning sensors 275; information based, atleast in part, on any of the above; any combination of the above; and soforth.

In some embodiments, obtaining first group of images (Step 610) and/orobtaining second group of images (Step 630) and/or obtaining a stream ofimages (Step 910) and/or receiving image data (Step 1110) may comprisereceiving input data using communication devices (such as communicationmodules 230, internal communication modules 440, external communicationmodules 450, and so forth). Examples of such input data may include:input data captured using one or more sensors; image data captured usingimage sensors, for example using image sensors 260; audio data capturedusing audio sensors, for example using audio sensors 250; motioninformation captured using motion sensors, for example using motionsensors 270; position information captured using positioning sensors,for example using positioning sensors 275; and so forth.

In some embodiments, obtaining first group of images (Step 610) and/orobtaining second group of images (Step 630) and/or obtaining a stream ofimages (Step 910) and/or receiving image data (Step 1110) may comprisereading input data from memory (such as memory units 210, shared memorymodules 410, and so forth). Examples of such input data may include:input data captured using one or more sensors; image data captured usingimage sensors, for example using image sensors 260; audio data capturedusing audio sensors, for example using audio sensors 250; motioninformation captured using motion sensors, for example using motionsensors 270; position information captured using positioning sensors,for example using positioning sensors 275; and so forth.

In some embodiments, analyzing image data, for example by Step 620and/or Step 660 and/or Step 720 and/or Step 750 and/or Step 920 and/orStep 930 and/or Step 1120, may comprise analyzing the image data toobtain a preprocessed image data, and subsequently analyzing the imagedata and/or the preprocessed image data to obtain the desired outcome.One of ordinary skill in the art will recognize that the followings areexamples, and that the image data may be preprocessed using other kindsof preprocessing methods. In some examples, the image data may bepreprocessed by transforming the image data using a transformationfunction to obtain a transformed image data, and the preprocessed imagedata may comprise the transformed image data. For example, thetransformed image data may comprise convolutions of the image data. Forexample, the transformation function may comprise image filters, such aslow-pass filters, high-pass filters, band-pass filters, all-passfilters, and so forth. In some examples, the transformation function maycomprise a nonlinear function. In some examples, the image data may bepreprocessed by smoothing the image data, for example using Gaussianconvolution, using a median filter, and so forth. In some examples, theimage data may be preprocessed to obtain a different representation ofthe image data. For example, the preprocessed image data may comprise: arepresentation of at least part of the image data in a frequency domain;a Discrete Fourier Transform of at least part of the image data; aDiscrete Wavelet Transform of at least part of the image data; atime/frequency representation of at least part of the image data; arepresentation of at least part of the image data in a lower dimension;a lossy representation of at least part of the image data; a losslessrepresentation of at least part of the image data; a time order seriesof any of the above; any combination of the above; and so forth. In someexamples, the image data may be preprocessed to extract edges, and thepreprocessed image data may comprise information based on and/or relatedto the extracted edges. In some examples, the image data may bepreprocessed to extract image features from the image data. Someexamples of such image features may comprise information based on and/orrelated to: edges; corners; blobs; ridges; Scale Invariant FeatureTransform (SIFT) features; temporal features; and so forth.

In some embodiments, analyzing image data, for example by Step 620and/or Step 660 and/or Step 720 and/or Step 750 and/or Step 920 and/orStep 930 and/or Step 1120, may comprise analyzing the image data and/orthe preprocessed image data using rules, functions, procedures,artificial neural networks, object detection algorithms, face detectionalgorithms, visual event detection algorithms, action detectionalgorithms, motion detection algorithms, background subtractionalgorithms, inference models, and so forth. Some examples of suchinference models may include: an inference model preprogrammed manually;a classification model; a regression model; a result of trainingalgorithms (such as machine learning algorithms and/or deep learningalgorithms) on training examples, where the training examples mayinclude examples of data instances, and in some cases, a data instancemay be labeled with a corresponding desired label and/or result; and soforth.

In some embodiments, identifying objects in the first group of images(Step 620) may comprise analyzing the image data and/or the preprocessedimage data obtain by Step 610 to identify a group of items, objects,faces, events, actions, and so forth, in the image data.

In some examples, identifying objects in the first group of images (Step620) may comprise using object detection algorithms to detect objects inthe image data obtained by Step 610 that match selected criteria. Someexamples of such object detection algorithms may include: appearancebased object detection algorithms, gradient based object detectionalgorithms, gray scale object detection algorithms, color based objectdetection algorithms, histogram based object detection algorithms,feature based object detection algorithms, machine learning based objectdetection algorithms, artificial neural networks based object detectionalgorithms, 2D object detection algorithms, 3D object detectionalgorithms, still image based object detection algorithms, video basedobject detection algorithms, and so forth.

In some examples, identifying objects in the first group of images (Step620) may comprise using face detection algorithms to detect facesmatching selected criteria in the image data obtained by Step 610, usingvisual event detection algorithms to detect events matching selectedcriteria in the image data obtained by Step 610, using action detectionalgorithms to detect actions matching selected criteria in the imagedata obtained by Step 610, and so forth.

In some examples, identifying objects in the first group of images (Step620) may comprise obtaining an indication of the object from a user. Forexample, object detection and/or recognition algorithms may be used tocompile a list of objects present in the image data obtained by Step610, the list may be presented to a user (for example, as a list oftextual descriptions of the objects, as a list of images of the objects,etc.), and the user may select an object from the list. In anotherexample, an image of the image data obtain by Step 610 may be presentedto a user, and user may point to an object, may mark a bounding boxaround an object, and so forth. In yet another example, a candidateobject may be presented to a user, and the user may indicate whetherthis object is acceptable or not. In some cases, the user may alsoindicate a type of the selected object, which may be used by Step 650 toselect a processing scheme.

In some examples, identifying objects in the first group of images (Step620) may comprise analyzing motion in the image data obtained by Step610, for example using motion segmentation algorithms, to identifysegments that correspond to a moving object. For example, when the imagesensor and the background are stationary, any motion in the image datamay correspond to moving objects.

In some embodiments, identifying regions in the second group of images(Step 640) may comprise identifying one or more regions of the imagedata and/or the preprocessed image data obtain by Step 630, for examplebased on the objects identified by Step 620 in the image data obtainedby Step 610. For example, the identified regions may comprise at least afirst region and a second region, where the first region may differ fromthe second region, may include all parts of the image data not includedin the second region, may include some parts of the image data notincluded in the second region, may have no common pixels with the secondregion, may have some common pixels with the second region, may includeall pixels of the second region, and so forth. In some examples, oneidentified region may comprise all pixels not included in otheridentified regions, not included in one other selected identifiedregion, not included in a group of selected identified regions, and soforth.

In some embodiments, identifying regions in the second group of images(Step 640) may comprise obtaining an indication of the region from auser. For example, alternative regions may be identified (for example asdescribed above) and presented to a user (for example as an overlay onan image of the image data obtained by Step 610), and the user mayselect some of the alternative regions. In some cases, the user may alsoindicate a type of the selected region, which may be used by Step 650 toselect a processing scheme.

In some embodiments, one of the regions identified by Step 640 in theimage data and/or the preprocessed image data obtained by Step 630 maycorrespond to a region of the image data obtained by Step 610 depictingall or part of an object identified by Step 620. For example, Steps 610and 630 may obtain images captured using a stationary image sensor 260with the same capturing parameters, and an identified region of theimage data obtained by Step 630 may comprise some or all of the pixelsdepicting the object in the image data obtained by Step 610, to abounding box that includes these pixels, and so forth. In anotherexample, Steps 610 and 630 may obtain images captured using a stationaryimage sensor 260 but with different capturing parameters, and anidentified region of the image data obtained by Step 630 may comprisesome or all of the pixels corresponding to the pixels depicting theobject in the image data obtained by Step 610 according to atransformation associated with the change in the capturing parameters,to a bounding box that includes these pixels, and so forth. In yetanother example, Steps 610 and 630 may obtain images captured using amoving image sensor 260, and an identified region of the image dataobtained by Step 630 may comprise at least some of the pixelscorresponding to the pixels depicting the object in the image dataobtained by Step 610 according to a transformation calculated accordingto the ego motion of the image sensor and/or to changes in the capturingparameters used, to a bounding box that includes these pixels, and soforth. In another example, Steps 610 and 630 may obtain images capturedusing different image sensors (for example, using image sensors 260included in apparatuses 200 a and 200 b), and an identified region ofthe image data obtained by Step 630 may comprise at least some of thepixels corresponding to the pixels depicting the object in the imagedata obtained by Step 610 according to a transformation associated withthe image sensors (such as a transformation calculated according to thefield of view of the image sensors, to the capturing parameters used bythe image sensors, to the position and/or orientation of the imagesensors, etc.), to a bounding box that includes these pixels, and soforth.

In some embodiments, selecting processing schemes (Step 650) maycomprise selecting a processing scheme of a plurality of alternativeprocessing schemes based on the objects identified by Step 620. Forexample, a processing scheme may be implemented as a formula, a computerprocedure, a computer function, and/or a computer program, and Step 650may select a formula, a computer procedure, a computer function, and/ora computer program of a plurality of alternative formulas, computerprocedures, computer functions, and/or computer programs based on theobjects identified by Step 620. In another example, a processing schememay be represented as a set of parameters (for example to a formula, acomputer procedure, a computer function, and/or a computer program), andStep 650 may select a set of parameters of a plurality of alternativesets of parameters based on the objects identified by Step 620. In yetanother example, a processing scheme may be implemented as an inferencemodel (such as a classifier, a regression model, an artificial neuralnetwork, a segmentation model, and so forth), and Step 650 may select aninference model based on the objects identified by Step 620. Someadditional examples of processing schemes may include ignoring theprocessed region, processing the region at a selected frame rate and/orfrequency, processing the region at a selected resolution, processingthe region to determine if the object identified by Step 620 is stillpresent in the region, processing the region only when the averageintensity of the region meets certain criteria (for example, is within aselected range of values), processing the region with a selectedprocessing scheme when the average intensity of the region meets certaincriteria (for example is within a selected range of values), and soforth.

In some embodiments, selecting processing schemes (Step 650) maycomprise determining a processing scheme according to training examples(for example by training a machine learning algorithm and/or a deeplearning algorithm and/or an artificial neural network to obtain aninference model from the training examples), and the training examplesmay be selected of a plurality of possible training examples.

In some embodiments, selecting processing schemes (Step 650) maycomprise using a rule to select a processing scheme of a plurality ofalternative processing schemes based on the objects identified by Step620. In some examples, a table may hold the alternative processingschemes (or identifiers of the processing schemes), and Step 650 mayaccess an entry of the table based on the objects identified by Step620. In some examples, the alternative processing schemes (oridentifiers of the processing schemes) may be stored in memory, and Step650 may fetch the selected processing scheme from the memory.

In some embodiments, processing the identified regions (Step 660) maycomprise processing regions identified by Step 640 using processingschemes selected by Step 650. For example, a selected processing schememay comprise a formula, and Step 660 may evaluate the formula using oneof the identified regions. In another example, a selected processingscheme may comprise a computer procedure, a computer function, and/or acomputer program, and Step 660 may execute the computer procedure,computer function, and/or computer program using the content of one ofthe identified regions as a parameter to the computer procedure,computer function, and/or computer program. In yet another example, aselected processing scheme may comprise an inference model, and Step 660may apply one of the identified regions to the inference model. Inanother example, according to the selected processing schemes, Step 660may ignore at least one identified region, process at least oneidentified region at a selected frame rate and/or frequency, process atleast one identified region at a selected resolution, process at leastone identified region to determine if an object identified by Step 620is still present in the region, process at least one identified regiononly when the average intensity of the region meets certain criteria(for example is within a selected range of values), process at least oneidentified region with a selected processing scheme when the averageintensity of the region meets certain criteria (for example is within aselected range of values), and so forth.

In some examples, Step 660 may also process the entire image dataobtained by Step 630 with a selected processing scheme, process at leastone selected region identified by Step 640 using a default processingscheme (which may be different from the processing schemes selected byStep 650), process image data obtained by Step 630 and not included inany regions identified by Step 640 using a selected processing scheme,process image data obtained by Step 630 and not included in one or moreselected regions of the regions identified by Step 640 using a selectedprocessing scheme, and so forth.

FIG. 7 illustrates an example of process 700 for selective use ofinference models. In some examples, process 700, as well as allindividual steps therein, may be performed by various aspects of:apparatus 200; server 300; cloud platform 400; computational node 500;and so forth. For example, process 700 may be performed by processingunits 220, executing software instructions stored within memory units210 and/or within shared memory modules 410. In this example, process700 may comprise: obtaining first group of images (Step 610); obtainingscene information (Step 720); obtaining inference models (Step 730);obtaining second group of images (Step 630); and processing the secondgroup of images using the inference models (Step 750). In someimplementations, process 700 may comprise one or more additional steps,while some of the steps listed above may be modified or excluded. Forexample, in some cases Steps 610 and/or 720 may be excluded from process700. In some implementations, one or more steps illustrated in FIG. 7may be executed in a different order and/or one or more groups of stepsmay be executed simultaneously and vice versa. For example, Step 630 maybe executed before, after and/or simultaneously with Step 610 and/orStep 720 and/or Step 730; Steps 720 and/or Step 730 may be executedafter and/or simultaneously with Step 610; Step 750 may be executedafter and/or simultaneously with Step 630; and so forth. Examples ofpossible execution manners of process 700 may include: continuousexecution, returning to the beginning of the process once the processnormal execution ends; periodically execution, executing the process atselected times; execution upon the detection of a trigger, whereexamples of such trigger may include trigger from a user, trigger fromanother process, trigger from an external device, etc.; any combinationof the above; and so forth.

In some embodiments, scene information obtained by Step 720 may compriseinformation related to data captured using one or more sensors from anenvironment. For example, the scene information may comprise:information related to the distribution of captured data (for example inthe form of frequencies at which different types of information arecaptured, in the form of a histogram, etc.), minimal levels captured,maximal levels captured, aggregated and/or statistical measurementsrelated to data captured over time, typical captured data instances,results of applying captured data to a clustering algorithm (such ask-means, spectral clustering, etc.), results of applying captured datato a dimensionality reduction algorithm (such as Principal ComponentAnalysis, Canonical Correlation Analysis, etc.), and so forth. Inanother example, the scene information may comprise information relatedto items and/or objects present and/or detected in the captured data.

In some embodiments, obtaining scene information (Step 720) may comprisegenerating the scene information, reading the scene information frommemory, receiving the scene information using communication devices(such as communication modules 230, internal communication modules 440,external communication modules 450, etc.), and so forth. In someembodiments, obtaining scene information (Step 720) may comprisegenerating scene information based, at least in part, on data capturedusing one or more sensors. Examples of such sensors may include audiosensors 250, image sensors 260, motion sensors 270, positioning sensors275, chemical sensors, temperature sensors, barometers, pressuresensors, proximity sensors, electrical impedance sensors, electricalvoltage sensors, electrical current sensors, and so forth.

In some embodiments, obtaining scene information (Step 720) may compriseanalyzing the images obtained by Step 610 to obtain the sceneinformation. In some examples, Step 720 may comprise selecting one ormore portions of the images, and the scene information may compriseinformation related to the selected portions. For example, image sensors260 may be stationary, a motion analysis of a video may be performed,and the portions of the video that has no or little movement may beselected. In another example, image gradients may be calculated, and theportions of the video that has high variance of gradients may beselected. In another example, face detector may be used to detect facesappearing in the one or more images, and portions of the imagescontaining faces may be selected.

In some examples, obtaining scene information (Step 720) may compriseextracting background from video obtained by Step 610, and the sceneinformation may comprise the extracted background of the environment,information related to the extracted background, and so forth. Examplesof algorithms for background extraction may include, taking the medianof the video, taking the median of the video after adjusting for egomotion of image sensors 260, taking the mean of the video, taking themean of the video after adjusting for ego motion of image sensors 260,taking the mode of the video, taking the mode of the video afteradjusting for ego motion of image sensors 260, and so forth.

In some examples, obtaining scene information (Step 720) may comprisedetecting objects in the images obtained by Step 610, and the sceneinformation may comprise identifying information of the detectedobjects, information related to the detected objects, positions at whichone or more objects were detected, frequencies at which differentobjects are detected, images of detected objects, and so forth. Forexample, the scene information may comprise identified properties of thedetected object, such as type, size, color, condition, and so forth. Insome cases, the scene information may comprise a mapping that specifiesfor different pixels and/or image regions the objects detected at thosepixels and/or regions, the frequencies at which objects are detected atthose pixels and/or regions, the frequencies at which specific objectsare detected at those pixels and/or regions, and so forth. Some examplesof object detection algorithms may include deep learning based objectdetection algorithms, appearance based object detection algorithms,image features based object detection algorithms, and so forth.

In some examples, obtaining scene information (Step 720) may comprisedetecting faces and/or recognizing faces in the images obtained by Step610, and the scene information may comprise identifying information ofthe detected and/or recognized faces, information related to thedetected and/or recognized faces, positions at which one or more faceswere detected, frequencies at which different faces are appearing,images of detected and/or recognized faces, and so forth. For example,the scene information may comprise identified properties of individualsappearing in the images, such as names, ages, gender, hair color,height, weight, and so forth. In some cases, the scene information maycomprise information related to people appearing regularly in theimages, for example of people appearing in more than a selected numberof images, over a selected time span, and so forth. In some cases, thescene information may comprise a mapping that specifies for differentpixels and/or image regions the faces detected at these pixels and/orregions, the frequencies at which faces are detected at these pixelsand/or regions, the frequencies at which specific faces are detected atthese pixels and/or regions, and so forth. Some examples of facedetection algorithms that may be used may include deep learning basedface detection algorithms, appearance based face detection algorithms,color based face detection algorithms, texture based face detectionalgorithms, shape based face detection algorithms, motion based facedetection algorithms, boosting based face detection algorithms, and soforth. Some examples of face recognition algorithms that may be used mayinclude deep learning based face recognition algorithms, appearancebased face recognition algorithms, color based face recognitionalgorithms, texture based face recognition algorithms, shape based facerecognition algorithms, motion based face recognition algorithms,boosting based face recognition algorithms, dimensionality reductionbased face recognition algorithms (such as eigenfaces, Fisherfaces,etc.), 3D face recognition algorithms, and so forth.

In some embodiments, obtaining scene information (Step 720) may comprisegenerating the scene information based, at least in part, on audio data,such as audio data captured using audio sensors 250 from an environmentof audio sensors 250. The audio data may be captured using audio sensors250, read from memory, received using communication devices (such ascommunication modules 230, internal communication modules 440, externalcommunication modules 450, etc.), and so forth.

In some examples, obtaining scene information (Step 720) may compriseidentifying characteristics of the ambient noise present in the audiodata captured, and the scene information may comprise the identifiedcharacteristics of the ambient noise, a model of the ambient noise,information related to the ambient noise, and so forth. For example, thenoise level may be monitored over time, and minimal noise level orhistogram of noise levels may be determined. In another example, typicalfrequencies of ambient noise may be identified, for example byclustering the frequencies present in the audio data when the noiselevel is below a selected threshold.

In some examples, obtaining scene information (Step 720) may compriseidentifying speakers in the audio data, and the scene information maycomprise the information related to the identified speakers. Forexample, voice models may be constructed for the identified speakers,and the scene information may comprise the voice models. In anotherexample, the scene information may comprise information related to thespeaking time of the speakers in the audio data, such as the totalspeaking time of each speaker, the total speaking time for all speakerscumulatively, histogram of the speaking times with respect to time inday, and so forth.

In some embodiments, obtaining inference models (Step 730) may compriseobtaining at least one inference model based, at least in part, on thescene information obtained by Step 720. For example, obtaining theinference model may comprise generating a first inference model for afirst scene information, a second inference model for a second sceneinformation, and so forth. In some examples, the scene information maybe generated by Step 720, received using one or more communicationdevices (such as communication modules 230, internal communicationmodules 440, external communication modules 450, etc.), read from memory(such as memory units 210, shared memory modules 410, etc.), and soforth.

In some examples, the scene information may comprise information and/orparameters that may be used by Step 730 to decide where to obtain theinference models from, which inference models to obtain, which parts ofthe obtained inference models to use, parameters for modifying theobtained inference models, where to read the inference models from,which inference models to read of a plurality of alternative inferencemodels stored in memory, which parts of the read inference models touse, parameters for modifying the read inference models, and so forth.

In some embodiments, obtaining inference models (Step 730) may compriseselecting an inference model of a plurality of alternative inferencemodels. For example, the plurality of alternative inference models maybe stored in memory (such as memory units 210, shared memory modules410, etc.), and the selection of the inference model may be based, atleast in part, on available information, such as the scene information.In some embodiments, obtaining inference models (Step 730) may compriseselecting one or more training examples, and training a machine learningalgorithm and/or a deep learning algorithm using the selected trainingexamples. For example, one or more training examples may be selected ofa plurality of alternative training examples. For example, the pluralityof alternative training examples may be stored in memory (such as memoryunits 210, shared memory modules 410, etc.), and the selection of thetraining examples may be based, at least in part, on the sceneinformation. In some embodiments, obtaining inference models (Step 730)may comprise selecting one or more components of an inference model, forexample by selecting one or more components of an inference model of aplurality of alternative components. For example, the plurality ofalternative components may be stored in memory (such as memory units210, shared memory modules 410, etc.), and the selection of thecomponents may be based, at least in part, on the scene information.

In some embodiments, at least part of the inference model obtained byStep 730 may be the result of training algorithms, such as machinelearning algorithms and deep learning algorithms, on training examples.The training examples may include examples of data instances, and insome cases, each data instance may be labeled with a correspondingdesired result. In some examples, the scene information may compriseinformation related to the training examples, and obtaining inferencemodels (Step 730) may comprise training algorithms based, at least inpart, on examples obtained using the scene information. In someexamples, the scene information may comprise labels for data instance,and the algorithms may be trained using these labels. In some examples,the scene information may comprise training examples, and the algorithmsmay be trained using these training examples. In some examples, thescene information may comprise information and/or parameters that may beused for obtaining training examples; and the algorithms may be trainedusing the training examples obtained based, at least in part, on theinformation and/or parameters included in the scene information. In someexamples, synthetic training examples may be generated, for example byselecting one or more parameters to a template of synthetic trainingexamples based, at least in part, on information included in the sceneinformation.

In some embodiments, at least part of the inference model obtained byStep 730 may comprise one or more artificial neural networks. In someembodiments, obtaining inference models (Step 730) may comprisegenerating one or more artificial neural network models, for example byselecting one or more parameters of an artificial neural network model,by selecting a portion of an artificial neural network model, byselecting one or more artificial neural network model of a plurality ofalternative artificial neural network models, by training an artificialneural network model on training examples, and so forth. In someembodiments, the inference models may comprise at least one of facedetector, face recognition model, object detector, motion detector,activity detector, gesture recognition model, image segmentation model,video segmentation models, speaker recognition model, speech recognitionmodel, audio segmentation model, a classifier, a regression model, asegmentation model, a combination of a plurality of inference models,and so forth.

In some embodiments, obtaining inference models (Step 730) may compriseobtaining an inference model based, at least in part, on historicalinformation stored in memory (such as historical information stored inmemory units 210 and/or shared memory units 410). For example, thehistorical information may comprise historical information from previousruns, from previous experience, and so forth. In some examples, thehistorical information may comprise scene information records, the sceneinformation obtained by Step 720 may be compared to the sceneinformation records, and the inference models may be based, at least inpart, on the comparison result. In some examples, the historicalinformation may comprise at least one rule for classifying the sceneinformation, the scene information may be classified using the at leastone rule, and the inference models may be based, at least in part, onthe classification result. In some examples, the historical informationmay comprise inference model records, and the inference models may bebased, at least in part, on the inference model records.

In some embodiments, obtaining inference models (Step 730) may comprisetransmitting to an external device the scene information obtained byStep 720, receiving at least part of the at least one inference model(for example from the external device and/or in response to thetransmitted scene information), and the received at least part of the atleast one inference model may be based on the transmitted sceneinformation. For example, the external device may receive thetransmitted scene information, generate and/or select inference modelsbased on the scene information as described above, and transmit thegenerated and/or selected inference models back to process 700.

In some embodiments, processing the second group of images using theinference models (Step 750) may comprise analyzing the images obtainedby Step 630 using the inference models obtain by Step 730.

In some embodiments, processing the second group of images using theinference models (Step 750) may comprise generating one or more resultsby applying input data (such as images and data obtained by Step 630) toinference models (for example, to the inference models obtained by Step730). In some examples, the inference model may comprise artificialneural networks, and the results may comprise at least part of theoutput of the artificial neural networks, information based on theoutput of the artificial neural networks, and so forth. In someexamples, the inference model may comprise classifiers, and the resultsmay comprise assignment of input data to one or more classes, assignmentof portions of the input data to one or more classes, information basedon the classifications, and so forth. In some examples, the inferencemodel may comprise regression models, and the results may comprisevalues assigned to the input data by the regression models, valuesassigned to portions of the input data by the regression models,information based on the values assigned by the regression models, andso forth. In some examples, the inference model may comprisesegmentation models, and the results may comprise information related toone or more segments identified in the input data.

In some examples, the input data may comprise audio data, the inferencemodel may comprise speaker recognition models, and the results maycomprise information related to speakers detected in the audio data bythe speaker recognition models. Examples of the information related tospeakers detected in the audio data may include information related tothe identities of the speakers, information related to the voices of thespeakers, information related to the content of the speech associatedwith the speakers, times at which the speakers were detected, audiosegments associated with the speakers, and so forth. In some examples,the input data may comprise audio data, the inference model may compriseone or more speech recognition models, and the results may compriseinformation related to speech detected in the audio data. Examples ofthe information related to speech may include information related tospeakers associated with the speech, information related to voicesassociated with the speech, times associated with the speech, audiosegments containing at least part of the speech, the content of thespeech (for example in a textual form), summary of the speech, topicsdiscussed in the speech, and so forth. In some examples, the input datamay comprise audio data, the inference model may comprise audiosegmentation models, and the results may comprise information related toaudio segments extracted from the audio data. In some examples, theinput data may comprise audio data, the inference model may comprisesource separation models, and the results may comprise informationrelated to audio sources identified in the audio data.

In some examples, the input data may comprise image data, the inferencemodel may comprise face detectors, and the results may compriseinformation related to faces appearing in the image data and detected bythe face detectors. Examples of the information related to the detectedfaces may include information related to the appearance of the faces,information related to the pose of the faces, information related tofacial expressions, image locations at which the faces were detected;times at which the faces were detected, images of the detected faces,and so forth. In some examples, the input data may comprise image data,the inference model may comprise face recognition models, and theresults may comprise information related to the identity of peopleappearing in the image data. In some examples, the input data maycomprise image data, the inference model may comprise object detectors,and the results may comprise information related to objects appearing inthe image data and detected by the object detectors. Examples of theinformation related to the detected objects may include object types,image locations at which the objects were detected, times at which theobjects were detected, images of the detected objects, and so forth. Insome examples, the input data may comprise image data, the inferencemodel may comprise one or more motion detectors, and the results maycomprise information related to motion detected in the image data. Insome examples, the input data may comprise image data, the inferencemodel may comprise activity detectors, and the results may compriseinformation related to activities detected in the image data. In someexamples, the input data may comprise image data, the inference modelmay comprise gesture recognition models, and the results may compriseinformation related to gestures detected in the image data. In someexamples, the input data may comprise image data, the inference modelmay comprise image segmentation models and/or video segmentation models,and the results may comprise information related to image and/or videosegments extracted from the image data.

In some examples, the input data may comprise image data, the inferencemodel may identify one or more pixels and/or voxels, and the results maycomprise the identified pixels and/or voxels and/or information relatedto the identified pixels and/or voxels. In some examples, the input datamay comprise image data, the inference model may identify portionsand/or regions of the image data, and the results may comprise theidentified portions and/or regions of the image data, informationrelated to the identified portions and/or regions of the image data, andso forth. In some examples, the input data may comprise image data, andthe results may comprise an association of values with portions and/orregions of the image data, for example in the form of a mapping thatmaps portions and/or regions of the image data to values.

In some examples, the input data may comprise information associatedwith a plurality of locations, the inference model may identifylocations based, at least in part, on the information associated with aplurality of locations, and the results may comprise the identified oneor more locations and/or information related to the identifiedlocations. In some examples, the input data may comprise informationassociated with locations and/or areas, the inference model may identifylocations and/or areas based, at least in part, on the associatedinformation, and the results may comprise the identified location and/orareas; information related to the identified locations and/or areas, andso forth. In some examples, the results may comprise an association ofvalues with locations and/or areas, for example in the form of a mappingthat maps locations and/or areas to values.

In some examples, the input data may comprise information associatedwith a plurality of times, the inference model may identify one or moretimes based, at least in part, on the information associated with aplurality of times, and the results may comprise the identified one ormore times and/or information related to the identified one or moretimes. In some examples, the input data may comprise informationassociated with one or more time ranges, the inference model mayidentify one or more times and/or one or more time ranges based, atleast in part, on the associated information, and the results maycomprise: the identified one or more times and/or one or more timeranges; information related to the identified one or more times and/orone or more time ranges; and so forth. In some examples, the results maycomprise an association of values with times and/or ranges of times, forexample in the form of a mapping that maps times and/or ranges of timesto values.

FIG. 8A is a schematic illustration of an example of environment 800 ofa room. In this example, environment 800 may comprise fan 802, blinkinglight source 804, non-blinking light source 806, window 808, televisionset 810, mirror 812, picture 814, and child 816.

In some examples, process 600 may capture image data of environment 800using Step 610 and Step 630, and Step 620 may detect fan 802 in theimage data captured by Step 610. Step 640 may identify regions in theimage data captured by Step 630 corresponding to the location at whichfan 802 was detected in the image data captured by Step 610. Based onthe detection of fan 802, Step 650 may select a processing scheme. Someexamples of the selected processing scheme may include ignoring motion,ignoring motion of fan 802, determining the speed and/or settings of fan802, and so forth. Step 660 may apply the selected processing scheme tothe regions identified by Step 640.

In some examples, process 700 may capture image data of environment 800using Step 610 and Step 630, and Step 720 may detect fan 802 in theimage data captured by Step 610 and generate scene informationspecifying the present of fan 802. Step 730 may select and/or receiveand/or generate an inference model based on the scene information. Forexample, the inference model may be configured to detect safety eventsassociated with fan 802, to determine the settings and/or speed of fan802, and so forth. Step 750 may process the image data captured by Step630 using the inference model. For example, Step 750 may process theimage data captured by Step 630 using the inference model to detectsafety events related to fan 802, such as a child coming near fan 802, achild playing with fan 802, a child sticking a finger into fan 802, andso forth. In some cases, process 700 may further comprise issuing awarning in response to the detection of the safety event, for examplethrough an audio speaker, through a text message to a care giver, and soforth.

In some examples, process 600 may capture image data of environment 800using Step 610 and Step 630, and Step 620 may detect blinking lightsource 804 and/or non-blinking light source 806 in the image datacaptured by Step 610. Step 640 may identify regions in the image datacaptured by Step 630 corresponding to the locations at which blinkinglight source 804 and/or non-blinking light source 806 were detected inthe image data captured by Step 610. Based on the detection of blinkinglight source 804 and/or non-blinking light source 806, Step 650 mayselect two processing schemes. Some examples of the selected processingscheme may include ignoring changes in the image data in a specifiedregion, ignoring blinking of blinking light source 804, determining ifblinking light source 804 and/or non-blinking light source 806 areactive, measuring the light intensity of blinking light source 804and/or non-blinking light source 806, and so forth. Step 660 may applyone of the selected processing schemes to the region identified by Step640 as corresponding to blinking light source 804, and another one ofthe selected processing schemes to the region identified by Step 640 ascorresponding to non-blinking light source 806.

In some examples, process 700 may capture image data of environment 800using Step 610 and Step 630, and Step 720 may detect blinking lightsource 804 and/or non-blinking light source 806 in the image datacaptured by Step 610 and generate scene information specifying thepresent of blinking light source 804 and/or non-blinking light source806. Step 730 may select and/or receive and/or generate inference modelsbased on the scene information. For example, the inference models may beconfigured to determining if blinking light source 804 and/ornon-blinking light source 806 are active, measure the light intensity ofblinking light source 804 and/or non-blinking light source 806, and soforth. Step 750 may process the image data captured by Step 630 usingthe inference models.

In some examples, process 600 may capture image data of environment 800using Step 610 and Step 630, and Step 620 may detect window 808 in theimage data captured by Step 610. Step 640 may identify regions in theimage data captured by Step 630 corresponding to the location at whichwindow 808 was detected in the image data captured by Step 610. Based onthe detection of window 808, Step 650 may select a processing scheme.Some examples of the selected processing scheme may include ignoringobjects and/or motion, ignoring objects and/or motion seen throughwindow 808, ignoring reflections on window 808, determining whether thewindow is open or close, and so forth. Step 660 may apply the selectedprocessing scheme to the regions identified by Step 640.

In some examples, process 700 may capture image data of environment 800using Step 610 and Step 630, and Step 720 may detect window 808 in theimage data captured by Step 610 and generate scene informationspecifying the present of window 808. Step 730 may select and/or receiveand/or generate an inference model based on the scene information. Forexample, the inference model may be configured to compensate forreflections on window 808, ignore objects and/or motion seen throughwindow 808, ignore reflections on window 808, determine whether thewindow is open or close, determine whether an object seen in the regionof the window is inside the room or outside the room, and so forth. Step750 may process the image data captured by Step 630 using the inferencemodel.

In some examples, process 600 may capture image data of environment 800using Step 610 and Step 630, and Step 620 may detect television set 810in the image data captured by Step 610. Step 640 may identify regions inthe image data captured by Step 630 corresponding to the location atwhich television set 810 was detected in the image data captured by Step610. Based on the detection of television set 810, Step 650 may select aprocessing scheme. Some examples of the selected processing scheme mayinclude ignoring objects and/or motion, ignoring objects and/or motionseen on the screen of television set 810, ignoring reflections on thescreen of television set 810, determining whether television set 810 isswitched on or off, identifying a channel and/or a program and/orcontent displayed on television set 810, and so forth. Step 660 mayapply the selected processing scheme to the regions identified by Step640. In another example, Step 640 may identify regions in the image datacaptured by Step 630 corresponding to positions that are in front oftelevision set 810 in the image data captured by Step 610, Step 650 mayselect a processing scheme that counts the number of people sitting inthe identified regions, and Step 660 may apply the selected processingscheme to the regions identified by Step 640.

In some examples, process 700 may capture image data of environment 800using Step 610 and Step 630, and Step 720 may detect television set 810in the image data captured by Step 610 and may generate sceneinformation specifying the present of television set 810. Step 730 mayselect and/or receive and/or generate an inference model based on thescene information. For example, the inference model may be configured toignore objects and/or motion seen on television set 810, ignorereflections on television set 810, determine whether television set 810is switched on or off, identify a channel and/or a program and/orcontent displayed on television set 810, count the number of peoplewatching television set 810, and so forth. Step 750 may process theimage data captured by Step 630 using the inference model.

In some examples, process 600 may capture image data of environment 800using Step 610 and Step 630, and Step 620 may detect mirror 812 in theimage data captured by Step 610. Step 640 may identify regions in theimage data captured by Step 630 corresponding to the location at whichmirror 812 was detected in the image data captured by Step 610. Based onthe detection of mirror 812, Step 650 may select a processing scheme.Some examples of the selected processing scheme may include ignoringobjects and/or motion, ignoring objects and/or motion seen on mirror812, ignoring reflections on mirror 812, and so forth. Step 660 mayapply the selected processing scheme to the regions identified by Step640.

In some examples, process 700 may capture image data of environment 800using Step 610 and Step 630, and Step 720 may detect mirror 812 in theimage data captured by Step 610 and generate scene informationspecifying the present of mirror 812. Step 730 may select and/or receiveand/or generate an inference model based on the scene information. Forexample, the inference model may be configured to ignore objects and/ormotion seen on mirror 812, ignore reflections on mirror 812, compensatefor reflections on mirror 812, determine whether an object seen in theregion of image corresponding to mirror 812 is a reflection or not, andso forth. Step 750 may process the image data captured by Step 630 usingthe inference model.

In some examples, process 600 may capture image data of environment 800using Step 610 and Step 630, and Step 620 may detect picture 814 in theimage data captured by Step 610. Step 640 may identify regions in theimage data captured by Step 630 corresponding to the location at whichpicture 814 was detected in the image data captured by Step 610. Basedon the detection of picture 814, Step 650 may select a processingscheme. Some examples of the selected processing scheme may includeignoring objects, ignoring stationary objects, ignoring objects depictedin picture 814, and so forth. Step 660 may apply the selected processingscheme to the regions identified by Step 640.

In some examples, process 700 may capture image data of environment 800using Step 610 and Step 630, and Step 720 may detect picture 814 in theimage data captured by Step 610 and generate scene informationspecifying the present of picture 814. Step 730 may select and/orreceive and/or generate an inference model based on the sceneinformation. For example, the inference model may be configured toignore objects depicted in picture 814, determine whether an object seenin the region of an image corresponding to picture 814 is part of thepicture or not, and so forth. For example, the inference model maycompare the image data captured by Step 630 to an image of picture 814as captured by Step 610. Step 750 may process the image data captured byStep 630 using the inference model.

In some examples, process 700 may capture image data of environment 800using Step 610 and Step 630, and Step 720 may detect child 816 in theimage data captured by Step 610 and generate scene informationspecifying the present of child 816. Step 730 may select and/or receiveand/or generate an inference model based on the scene information. Forexample, the inference model may be configured to detect safety eventsassociated with children, to identify and/or summarize the activities ofchild 816, and so forth. Step 750 may process the image data captured byStep 630 using the inference model.

In some examples, process 700 may capture image data of an environmentusing Step 610 and Step 630, and Step 720 may detect a pet in the imagedata captured by Step 610 and generate scene information specifying thepresent of the pet. Step 730 may select and/or receive and/or generatean inference model based on the scene information. For example, theinference model may be configured to detect safety events associatedwith the pet, to identify and/or record the activities of the pet, todetermine a state associated with the pet, and so forth. Step 750 mayprocess the image data captured by Step 630 using the inference model.

FIG. 8B is a schematic illustration of an example of environment 820 ofa yard. In this example, environment 820 may comprise swimming pool 822,tree 824, clouds 826, sky 828, and yard surface 830.

In some examples, process 600 may capture image data of environment 820using Step 610 and Step 630, and Step 620 may detect swimming pool 822in the image data captured by Step 610. Step 640 may identify regions inthe image data captured by Step 630 corresponding to the location atwhich swimming pool 822 was detected in the image data captured by Step610. Based on the detection of swimming pool 822, Step 650 may select aprocessing scheme. Some examples of the selected processing scheme mayinclude ignoring texture and/or motion, ignoring texture and/or motionof water in swimming pool 822, ignoring reflections on the water surfaceof swimming pool 822, correcting the image for refraction due to thewater in swimming pool 822, and so forth. Step 660 may apply theselected processing scheme to the regions identified by Step 640.

In some examples, process 700 may capture image data of environment 820using Step 610 and Step 630, and Step 720 may detect swimming pool 822in the image data captured by Step 610 and generate scene informationspecifying the present of swimming pool 822. Step 730 may select and/orreceive and/or generate an inference model based on the sceneinformation. For example, the inference model may be configured todetect safety events associated with swimming pool 822, detect drowningin swimming pool 822, identify unsupervised use of swimming pool 822 bya child, ignore texture and/or motion, ignore texture and/or motion ofwater in swimming pool 822, ignore reflections on the water surface ofswimming pool 822, correct refractions due to the water in swimming pool822, and so forth. Step 750 may process the image data captured by Step630 using the inference model. In another example, based on sceneinformation constructed by Step 720 in response to the detection ofchild 816 in the house and swimming pool 822 in the yard, Step 730 mayselect and/or receive and/or generate an inference model to identifyunsupervised use of swimming pool 822 by a child.

In some examples, process 600 may capture image data of environment 820using Step 610 and Step 630, and Step 620 may detect tree 824 in theimage data captured by Step 610. Step 640 may identify regions in theimage data captured by Step 630 corresponding to the location at whichtree 824 was detected in the image data captured by Step 610. Based onthe detection of tree 824, Step 650 may select a processing scheme. Someexamples of the selected processing scheme may include ignoring textureand/or motion, ignoring texture and/or motion of leafs of tree 824, andso forth. Step 660 may apply the selected processing scheme to theregions identified by Step 640.

In some examples, process 700 may capture image data of environment 820using Step 610 and Step 630, and Step 720 may detect tree 824 in theimage data captured by Step 610 and generate scene informationspecifying the present of tree 824. Step 730 may select and/or receiveand/or generate an inference model based on the scene information. Forexample, the inference model may be configured to detect safety eventsassociated with tree 824, identify unsupervised climbing of a child ontree 824, ignore texture and/or motion, ignore texture and/or motion ofleafs of tree 824, and so forth. Step 750 may process the image datacaptured by Step 630 using the inference model. In another example,based on scene information constructed by Step 720 in response to thedetection of child 816 in the house and tree 824 in the yard, Step 730may select and/or receive and/or generate an inference model to identifyunsupervised climbing of a child on tree 824.

In some examples, process 600 may capture image data of environment 820using Step 610 and Step 630, and Step 620 may detect clouds 826 and/orsky 828 in the image data captured by Step 610. Step 640 may identifyregions in the image data captured by Step 630 corresponding to thelocation at which clouds 826 and/or sky 828 were detected in the imagedata captured by Step 610. Based on the detection of clouds 826 and/orsky 828, Step 650 may select a processing scheme. Some examples of theselected processing scheme may include ignoring texture and/or motion,ignoring texture and/or motion of clouds, ignoring birds, ignoringairplanes, detecting birds, detecting airplanes, and so forth. Step 660may apply the selected processing scheme to the regions identified byStep 640.

In some examples, process 700 may capture image data of environment 820using Step 610 and Step 630, and Step 720 may detect clouds 826 and/orsky 828 in the image data captured by Step 610 and generate sceneinformation specifying the present of clouds 826 and/or sky 828. Step730 may select and/or receive and/or generate an inference model basedon the scene information. For example, the inference model may beconfigured to ignore texture and/or motion, ignore texture and/or motionof clouds, ignore birds, ignore airplanes, detect birds, detectairplanes, count birds, count airplanes, and so forth. Step 750 mayprocess the image data captured by Step 630 using the inference model.

In some examples, process 600 may capture image data of environment 820using Step 610 and Step 630, and Step 620 may detect yard surface 830 inthe image data captured by Step 610. Step 640 may identify regions inthe image data captured by Step 630 corresponding to the location atwhich yard surface 830 was detected in the image data captured by Step610. Based on the detection of yard surface 830, Step 650 may select aprocessing scheme. For example, the selected processing scheme mayinclude ignoring texture and/or motion. In another example, the yardsurface 830 may be covered with grass, and the selected processingscheme may include ignoring texture and/or motion of the grass. In yetanother example, the yard surface 830 may be covered with sand, and theselected processing scheme may include ignoring the texture of the sand.Step 660 may apply the selected processing scheme to the regionsidentified by Step 640.

In some examples, process 700 may capture image data of environment 820using Step 610 and Step 630, and Step 720 may detect yard surface 830 inthe image data captured by Step 610 and generate scene informationspecifying the present of yard surface 830. Step 730 may select and/orreceive and/or generate an inference model based on the sceneinformation. For example, the inference model may be configured toignore texture and/or motion of the yard surface, to detect intruders,and so forth. Step 750 may process the image data captured by Step 630using the inference model.

FIG. 9 illustrates an example of a process 900 for facilitating learningof visual events. In some examples, process 900, as well as allindividual steps therein, may be performed by various aspects of:apparatus 200; server 300; cloud platform 400; computational node 500;and so forth. For example, process 900 may be performed by processingunits 220, executing software instructions stored within memory units210 and/or within shared memory modules 410. In this example, process900 may comprise: obtaining a stream of images (Step 910); obtainingpoints in time (Step 920); for a point in time, identifying eventsrelated to an activity and preceding the point in time (Step 930);providing information about the association of events and activities(Step 940); obtaining feedback (Step 950); and obtaining event detectionrule (Step 960). In some implementations, process 900 may comprise oneor more additional steps, while some of the steps listed above may bemodified or excluded. For example, in some cases Steps 940 and/or 950and/or 960 may be excluded from process 900. In some implementations,one or more steps illustrated in FIG. 9 may be executed in a differentorder and/or one or more groups of steps may be executed simultaneouslyand vice versa. For example, Step 920 may be executed before, afterand/or simultaneously with Step 910, and so forth. In someimplementations, process 900 may repeat Step 930 for a plurality ofpoints in time. Examples of possible execution manners of process 900may include: continuous execution, returning to the beginning of theprocess once the process normal execution ends; periodically execution,executing the process at selected times; execution upon the detection ofa trigger, where examples of such trigger may include trigger from auser, trigger from another process, trigger from an external device,etc.; any combination of the above; and so forth.

In some embodiments, obtaining a stream of images (Step 910) maycomprise obtaining a stream of images captured using at least one imagesensor from an environment, for example as described above. In someembodiments, obtaining a stream of images (Step 910) may comprise, inaddition or alternatively to obtaining the stream of images, obtainingother inputs, for example as described above.

In some embodiments, obtaining points in time (Step 920) may compriseobtaining points in time associated with at least one activity, forexample by analyzing input data to determine the points in time. Thepoints in time may correspond to specific images and/or group of imagesin the stream of images obtained by Step 910.

In some examples, the input data may comprise the stream of imagesobtained by Step 910, Step 920 may analyze the stream of images toidentify images and/or group of images in the stream of images, and theidentified images and/or group of images may define the points in time.For example, Step 920 may analyze the stream of images to identifyimages and/or group of images depicting at least one of a person beinginjured, an accident occurring, people running, people running from aplace, people running towards a place, etc., and the points in time maycorrespond to the identified images and/or group of images. In someexamples, the stream of images may be analyzed using an object detectionalgorithms, and the points in time may correspond to the first and/orlast and/or selected appearances of objects in the stream of images. Insome examples, the stream of images may be analyzed using an eventdetection algorithms and/or action detection algorithm, and the pointsin time may correspond to the detected events and/or actions. In someexamples, the stream of images may be analyzed using a machine learningalgorithm and/or an artificial neural net trained to detected selecteditems in the stream of images and/or selected points in time accordingbased on the stream of images. For example, the machine learningalgorithm and/or the artificial neural net may be trained using trainingexamples, and a training example may comprise sample images along withlabels corresponding to items in the sample images and/or points in timecorresponding to the images.

In some examples, the input data may comprise input data that issynchronized with the stream of images obtained by Step 910, Step 920may analyze the input data to identify the points in time, and in somecases Step 920 may further determine images and/or group of images ofthe stream of images corresponding to the identified points in time. Insome examples, the input data may comprise input data that issynchronized with a second clock and the stream of images obtained byStep 910 may also be synchronized with a clock, Step 920 may analyze theinput data to identify the points in time, and in some cases Step 920may further determine images and/or group of images of the stream ofimages corresponding to the identified points in time based on the timeof the two clocks.

In some examples, the input data may comprise audio data, Step 920 mayanalyze the audio data to identify the points in time, and in somecases, Step 920 may further determine images and/or group of images ofthe stream of images corresponding to the identified points in time asdescribed above. For example, Step 920 may analyze the audio data todetect in the audio data at least of one of a verbal warning, a rebuke,a yelling, a call for help, an alarm sound, etc., and the points in timemay correspond to the detected items in the audio data. In someexamples, the audio data may be processed using a speech to textalgorithm, and the resulting textual information may be analyzed using anatural language processing algorithm to detect textual informationcorresponding to items in the audio data and/or to identify the pointsin time directly. In some examples, the pitch of the audio data may beanalyzed, for example using a threshold, to identify the points in timedirectly. In some examples, the audio data may be analyzed using aspeaker diarization algorithms and/or speaker recognition algorithms,and the detection of the points in time may be based on the analysisresults. In some examples, the audio data may be analyzed using amachine learning algorithm and/or an artificial neural net trained todetected selected items in the audio data and/or selected points intime. For example, the machine learning algorithm and/or the artificialneural net may be trained using training examples, and a trainingexample may comprise sample audio data along with labels correspondingto items and/or points in time in the sample audio data.

In some examples, the input data may comprise data stream from anexternal electronic device, Step 920 may analyze the data stream toidentify the points in time, and in some cases, Step 920 may furtherdetermine images and/or group of images of the stream of imagescorresponding to the identified points in time as described above. Someexamples of such external electronic device may include a computerizeddevice, a telephone, a smartphone, a tablet, a personal computer, atelevision set, an electronic media center, a car, an autonomous car,and so forth. For example, the data stream may comprise an indicationand/or details of a phone call, and Step 920 may identify points in timecorresponding to the phone calls, to some selected phone calls, to phonecalls to an emergency center, to phone calls to a delivery service, tophone calls to restaurants, to phone calls to car service, and so forth.In another example, the data stream may comprise an indication and/ordetails of a usage of an application and/or a software product, and Step920 may identify points in time corresponding to the usage of theapplication and/or the software product, to selected usages of theapplication and/or the software product, to a usage of the applicationand/or the software product for accessing selected information, to acontact to a with an emergency center and/or service using theapplication and/or the software product, to an ordering of a deliveryusing the application and/or the software product, to an ordering offood using the application and/or the software product, to an orderingof products using the application and/or the software product, to anordering of a car service using the application and/or the softwareproduct, and so forth. In yet another example, the data stream maycomprise an indication and/or details of accesses to a web and/or onlineservices, and Step 920 may identify points in time corresponding to theaccesses to the web and/or online services, to selected accesses to theweb and/or online services, to accesses to selected web and/or onlineservices, to an access to a web and/or online service of an emergencyservice, to an access to a web and/or online service of a deliveryservice, to an access to a web and/or online service of a restaurant, toan access to a web and/or online service of a car service, and so forth.In yet another example, the data stream may comprise one or more pointsin time identified by an external device and/or an external service.

In some examples, the input data may comprise inputs from a user, Step920 may analyze the inputs to identify the points in time, and in somecases, Step 920 may further determine images and/or group of images ofthe stream of images corresponding to the identified points in time asdescribed above. Some examples of such inputs from users may include keypresses, voice commands, hand gestures, and so forth. For example, aninput from a user may signal Step 920 that the current time should berecognized as a point in time, that a time corresponding to a previousactivity should be recognized as a point in time, and so forth.

In some embodiments, identifying events related to an activity andpreceding a point in time (Step 930) may be repeated for one, some orall points in time obtained by Step 920. In some examples, identifyingevents related to an activity and preceding a point in time (Step 930)may comprise analyzing the stream of images obtained by Step 910 toidentify one or more events related to at least one activity associatedwith the point in time and preceding the point in time. For example, apoint in time may correspond to specific images and/or group of imagesin the stream of images obtained by Step 910, and Step 930 may analyzeimages preceding the images and/or group of images that corresponds tothe point in time to identify one or more events related to at least oneactivity associated with the point in time.

In some examples, the point in time may correspond to a person beinginjured and/or to an accident occurring, for example to a depiction of aperson being injured and/or of an accident occurring in the stream ofimages, and Step 930 may analyze the images preceding the injury and/orthe accident to identify events leading to the injury and/or theaccident, such as a person running, a person using a piece of equipmentwithout proper safety equipment, hazardous conditions in theenvironment, malfunction equipment, and so forth. In some examples, thepoint in time may correspond to people running, people running from aplace, people running towards a place, etc., for example to a depictionof people running in the stream of images, and Step 930 may analyze theimages preceding the running to identify an event causing the people torun, such as an injury, an accident, a safety event, and so forth.

In some examples, the point in time may correspond to a verbal warningand/or a rebuke and/or a yelling, for example to a verbal warning and/ora rebuke and/or a yelling detected by Step 920 in an audio data, andStep 930 may analyze the images preceding the time of the verbal warningand/or the rebuke and/or the yelling to identify an event leading to theverbal warning and/or the rebuke and/or the yelling, such as aninappropriate behavior, a safety related event, a person failing toperform a task, and so forth. In some examples, the point in time maycorrespond to a call for help and/or a cry of pain, for example to acall for help and/or a cry of pain detected by Step 920 in an audiodata, and Step 930 may analyze the images preceding the time of the callfor help and/or the cry of pain to identify an event leading to the callfor help and/or the cry of pain, such as an injury, an accident, asafety event, and so forth. In some examples, the point in time maycorrespond to an alarm sound, for example to an alarm sound detected byStep 920 in an audio data, and Step 930 may analyze the images precedingthe time of the alarm sound to identify an event leading to the alarm,such as an injury, an accident, a safety event, and so forth.

In some examples, the point in time may correspond to a phone call (suchas a phone call to an emergency center, to a delivery service, to arestaurant, to a car service, etc.), and Step 930 may analyze the imagespreceding the time of the phone call to identify events leading to thephone call, such as an emergency situation, an injury, an accident, asafety related event, opening of a refrigerator, opening of a door,grabbing of a bag and/or a coat, existing a room and/or a house, and soforth. In some examples, the point in time may correspond to a usage ofan application and/or a software product (such as usage of theapplication and/or the software product to access selected information,to contact an emergency center and/or service, to order a delivery, toorder food, to order products, to order a car service, etc.), and Step930 may analyze the images preceding the time of the usage of theapplication and/or the software product to identify events leading tothe usage of the application and/or the software product, such as anemergency situation, an injury, an accident, a safety related event,opening of a refrigerator, opening of a door, grabbing of a bag and/or acoat, existing a room and/or a house, and so forth. In some examples,the point in time may correspond to an access to a web service (such asaccessing a web service to obtain information, to contact an emergencyservice, to order a delivery, to order from a restaurant, to order a carservice, etc.), and Step 930 may analyze the images preceding the timeof the access to the web service to identify events leading to theaccess to the web service, such as an emergency situation, an injury, anaccident, a safety related event, opening of a refrigerator, opening ofa door, grabbing of a bag and/or a coat, existing a room and/or a house,and so forth.

In some examples, the point in time may correspond to an input from auser (for example, in the form of a key press, a voice command, a handgesture, etc.), and Step 930 may analyze the images preceding the timeof the user input to identify events associated and/or leading to theuser input.

In some examples, Step 930 may analyze the images preceding the point intime using object detection algorithms to detect events that comprise apresent of a selected object in an environment. In some examples, Step930 may analyze the images preceding the point in time using an eventdetection algorithms and/or action detection algorithm to detect eventsthat comprise the occurrence of a selected event and/or the performanceof a selected action. In some examples, Step 930 may analyze the imagespreceding the point in time using a machine learning algorithm and/or anartificial neural net trained to detected selected events in a stream ofimages. For example, the machine learning algorithm and/or theartificial neural net may be trained using training examples, and atraining example may comprise sample images along with labelscorresponding to events in the sample images.

In some embodiments, providing information about the association ofevents and activities (Step 940) may comprise providing informationrelated to points in time obtained by Step 920 and/or to activitiesidentified by Step 920 and/or to events identified by Step 930. In someexamples, one or more alternatives associations of activities and/orevents and/or points in time may be provided to the user, and in somecases the user may select associations out of the alternativeassociations. In some examples, the information may be providedvisually, for example using a graphical user interface, using a website, using a display system, using an augmented reality system, using avirtual reality system, in a printed form, and so forth. For example,Step 940 may visually present to a user a graph depicting associationsof activities and/or events and/or points in time, or a table listingthe associations. In some cases, the activities and/or events and/orpoints in time may be presented using images depicting the activitiesand/or events and/or points in time, for example using images selectedfrom the stream of images obtained by Step 910. In some cases, theactivities and/or events and/or points in time may be presented astextual information describing the activities and/or events and/orpoints in time. In some examples, the information may be providedaudibly, for example through audio speakers, using head set, and soforth. For example, a list of associations of activities and/or eventsand/or points in time may be read aloud. In another example, activitiesand/or events and/or points in time may be described literally, forexample by taking textual information describing the activities and/orevents and/or points in time, and converting it to audible output usingtext to speech algorithm.

In some embodiments, obtaining feedback (Step 950) may compriseobtaining input related to information provided by Step 940. In someexamples, the input may be entered through a graphical user interface,through a web site, using a keyboard and/or a mouse and/or a touch padand/or a touch screen, using a microphone as voice input and/or voicecommands, using a camera as hand gestures, and so forth. In someexamples, the feedback may comprise indications from the user regardingthe associations of activities and/or events and/or points in time. Forexample, the user may reject some associations, may correct and/orchange some associations, may add associations, may modify informationrelated to activities and/or events and/or points in time, and so forth.For example, the user may modify a point in time by entering a differenttime index, by selecting an image from the stream of images obtained byStep 910, and so forth. In another example, the user may modify anactivity and/or event by pointing to images and/or regions within imagesthat depict the desired activity and/or event. In another example, theuser may modify an association of activities and/or events and/or pointsin time by removing some of the activities and/or events and/or pointsin time, by selecting activities and/or events and/or points in time tobe added to the association from a plurality of alternative activitiesand/or events and/or points in time, by switching an activity and/or anevent and/or a point in time with a different activity and/or eventand/or point in time (for example by selecting the activity and/or eventand/or point in time to be replaced and/or by selecting the new activityand/or event and/or point in time from a plurality of alternativeactivities and/or events and/or points in time).

In some embodiments, obtaining event detection rule (Step 960) maycomprise obtaining one or more event detection rules configured toanalyze images to detect one or more events, for example based on theevents identified by Step 930. In some examples, Step 960 may select oneor more event detection rules of a plurality of alternative eventdetection rules, for example based on the events identified by Step 930.For example, a data structure containing records, where each recordcontaining an event type identifier and a set of alternative eventdetection rules, may be accessed according to the type of the eventsidentified by Step 930 to select sets of alternative event detectionrules. In another example, a rule for selecting event detection rules ofa plurality of alternative event detection rules according to eventsidentified by Step 930 may be used. In some examples, Step 960 may traina machine learning algorithm using a plurality of training examples toobtain the one or more event detection rules. Some examples of suchmachine learning algorithms may include deep learning algorithms,trainable artificial neural network, support vector machines, randomforest, trainable classifiers, trainable object detectors, trainableevent detectors, trainable action detectors, and so forth. In someexamples, the plurality of training examples may be based on the eventsidentified by Step 930. For example, the plurality of training examplesmay include at least part of the events identified by Step 930. Inanother example, at least some of the plurality of training examples maybe selected from a plurality of alternative training examples and/orfrom a plurality of alternative sets of training examples based on theevents identified by Step 930, for example using a selection rule.

In some embodiments, Step 960 may further base the one or more eventdetection rules on the feedback obtained by Step 950. For example, thefeedback may include a selection of events out of the events identifiedby Step 930, and Step 960 may base on the event detection rules on theselected events, possibly ignoring the events that were not selected. Inanother example, the feedback may include an assignment of weights toevents, and Step 960 may base on the event detection rules on theweights, for example by assigning weights to at least some trainingexamples used by Step 960 (as described above) according to thefeedback. In yet another example, the feedback may split the eventsidentified by Step 930 into groups (for example by associating theevents with different activities), and Step 960 may base different eventdetection rules on different groups of events.

FIG. 10A is a schematic illustration of an example image 1000 capturedby an apparatus, such as apparatus 200. In this example, image 1000 maycomprise person 1002 being injured. FIG. 10B is a schematic illustrationof an example image 1010 captured by an apparatus, such as apparatus200. In this example, image 1010 may comprise persons 1012 and 1014running. FIG. 10C is a schematic illustration of an example image 1020captured by an apparatus, such as apparatus 200. In this example, image1020 may comprise person 1022 using phone 1024. FIG. 10D is a schematicillustration of an example image 1030 captured by an apparatus, such asapparatus 200. In this example, image 1030 may comprise person 1032using computerized device 1034 (such as a smartphone, a tablet, apersonal computer, etc.). Process 900 may obtain images 1000 and/or 1010and/or 1020 and/or 1030 using Step 910.

In some examples, person 1002 and person 1012 may be the same person,and image 1010 may precede image 1000 in time. In such case, Step 920may identify a point in time corresponding to image 1000 and recognizethat person 1002 is being injured in image 1000, Step 930 may identifythat person 1002 is running in image 1010 and deduce that the running inimage 1010 leads to the injury in image 1000, Step 960 may generate anevent detector configured to detect when people (or selected people) arerunning, and the event detector may be used to analyze future images andwarn when people (or the selected people) are running, for example inorder to prevent further injuries.

In some examples, image 1000 may precede image 1010 in time. In suchcase, Step 920 may identify a point in time corresponding to image 1010and recognize that persons 1012 and 1014 are running in image 1010, Step930 may identify that person 1002 is being injured in an accidentoccurring in image 1000 and deduce that the accident in image 1000caused the people to run in image 1010 (for example toward person 1002,to help person 1002, away from the accident, etc.), Step 960 maygenerate an event detector configured to detect people being injuredand/or accidents, and the event detector may be used to analyze futureimages and detect emergency situations.

In some examples, image 1000 may precede image 1020 in time. In suchcase, Step 920 may identify a point in time corresponding to image 1020and recognize that person 1022 is using phone 1024 to call an emergencycenter (for example, by analyzing audio and/or by receiving dialinginformation from phone 1024), Step 930 may identify that person 1002 isbeing injured in an accident occurring in image 1000 and deduce that theaccident in image 1000 caused person 1022 to call the emergency centerin image 1020, Step 960 may generate an event detector configured todetect people being injured and/or accidents, and the event detector maybe used to analyze future images and detect emergency situations and/orto automatically contact an emergency center.

In some examples, image 1000 may precede image 1030 in time. In suchcase, Step 920 may identify a point in time corresponding to image 1030and recognize that person 1032 is using computerized device 1034 tocontact an emergency center (for example by receiving usage informationfrom computerized device 1034), Step 930 may identify that person 1002is being injured in an accident occurring in image 1000 and deduce thatthe accident in image 1000 caused person 1032 to contact the emergencycenter in image 1030, Step 960 may generate an event detector configuredto detect people being injured and/or accidents, and the event detectormay be used to analyze future images and detect emergency situationsand/or to automatically contact an emergency center.

In some examples, images depicting person 1022 looking for food mayprecede image 1020 in time. In such case, Step 920 may identify a pointin time corresponding to image 1020 and recognize that person 1022 isusing phone 1024 to call a food delivery service or to make areservation at a restaurant (for example, by analyzing audio and/or byreceiving dialing information from phone 1024), Step 930 may identifythat person 1022 looked for food in the images preceding the usage ofthe phone and deduce that this lead to the usage of the phone, Step 960may generate an event detector configured to detect people looking forfood, and the event detector may be used to analyze future images anddetect people looking for food, for example to automatically order adelivery or suggest a restaurant.

In some examples, images depicting person 1032 looking for food mayprecede image 1030 in time. In such case, Step 920 may identify a pointin time corresponding to image 1030 and recognize that person 1032 isusing computerized device 1034 to order a delivery or to make areservation at a restaurant (for example by receiving usage informationfrom computerized device 1034), Step 930 may identify that person 1032looked for food in the images preceding the usage of the computerizeddevice and deduce that this lead to the usage of the computerizeddevice, Step 960 may generate an event detector configured to detectpeople looking for food, and the event detector may be used to analyzefuture images and detect people looking for food, for example toautomatically order a delivery or suggest a restaurant.

In some examples, image 1020 or image 1030 may precede in time images ofperson 1022 or person 1032 (respectively) leaving a house. In such case,Step 920 may identify a point in time corresponding to the leaving ofthe house, Step 930 may identify that the person leaving the houseordered a car service or reserved a place at a restaurant (for exampleusing phone 1024 in image 1020 or using computerized device 1034 inimage 1030), Step 960 may generate an event detector configured todetect people ordering a car service or reserving place at a restaurant,and the event detector may be used to analyze future images and predictwhen people are about to leave the house.

FIG. 11 illustrates an example of process 1100 for collectinginformation about repeated behavior. In some examples, process 1100, aswell as all individual steps therein, may be performed by variousaspects of: apparatus 200; server 300; cloud platform 400; computationalnode 500; and so forth. For example, process 1100 may be performed byprocessing units 220, executing software instructions stored withinmemory units 210 and/or within shared memory modules 410. In thisexample, process 1100 may comprise: receiving image data (Step 1110);identifying instances of a repeated activity in the image data (Step1120); determining properties of the repeated activity (Step 1130); andproviding information based on the determined properties (Step 1140). Insome implementations, process 1100 may comprise one or more additionalsteps, while some of the steps listed above may be modified or excluded.For example, in some cases Steps 1110 and/or Step 1120 and/or Step 1140may be excluded from process 1100. In some implementations, one or moresteps illustrated in FIG. 11 may be executed in a different order and/orone or more groups of steps may be executed simultaneously and viceversa. For example, Step 1120 and/or Step 1130 may be executed afterand/or simultaneously with Step 1110, Step 1130 may be executed before,after and/or simultaneously with Step 1120, Step 1140 may be executedafter and/or simultaneously with Step 1130, and so forth. Examples ofpossible execution manners of process 1100 may include: continuousexecution, returning to the beginning of the process once the processnormal execution ends; periodically execution, executing the process atselected times; execution upon the detection of a trigger, whereexamples of such trigger may include trigger from a user, trigger fromanother process, trigger from an external device, etc.; any combinationof the above; and so forth.

In some embodiments, receiving image data (Step 1110) may compriseobtaining a stream of images captured using one or more image sensorsfrom an environment, for example as described above. In someembodiments, receiving image data (Step 1110) may comprise, in additionor alternatively to obtaining the stream of images, obtaining otherinputs, for example as described above, for example as described above.

In some embodiments, receiving image data (Step 1110) may compriseobtaining information based on image data, for example based on a streamof images captured by an apparatus, such as apparatus 200. For example,a stream of images may be captured and analyzed by another processand/or an external device, and Step 1110 may receive data containingresults of the analysis. Such results may include information related toitems detected in the image data. For example, an item may include anobject or a person, and the information may comprise a type of anobject, identity of an item, location of an item, times at which an itemappears, time an item first or last appears in the image data, otherproperties of an item, and so forth. In another example, an item mayinclude events, activities, behaviors, and so forth, and the informationmay comprise properties of the item, information related to the item,type of item, time the item occurred, identities of people involved,objected used, locations, and so forth.

In some embodiments, identifying instances of a repeated activity in theimage data (Step 1120) may comprise analyzing information received byStep 1110 to identify instances of a repeated activity and/or behavior.In some examples, the information may be analyzed to identify instancesof a repeated activity and/or behavior of a selected person or of aselected group of people. In some examples, Step 1120 may be repeated toidentify instances of a repeated activity and/or behavior of differentpersons or different groups of people. In some examples, the informationmay be analyzed to identify instances of a repeated activity and/orbehavior (possibly of a selected person or a selected group of people)that repeat at least a select minimal number of times (such as twotimes, three times, five times, ten times, one hundred times, and soforth).

In some embodiments, information received by Step 1110 may compriseimage data, and Step 1120 may analyze the image data to identifyactivities and/or behaviors in the image data. In some examples, machinelearning and/or deep learning algorithms trained to detect activitiesand/or behaviors in images, possibly together with properties of theactivities and/or behaviors, may be used. For example, the machinelearning and/or deep learning algorithms may be trained using trainingimages, and the training images may be labeled according to whichactivities and/or behaviors appearing in the images, and/or according toproperties of the activities and/or behaviors appearing in the images.In another example, face recognition algorithms may be used to determinewhich person or group of people are involved in an activity and/orbehavior. In yet another example, object detection algorithms may beused to identify which objects are used or involved in the activity. Inanother example, the location of the activity and/or behavior may bedetermined based on the location the activity and/or behavior appears inthe image data, possibly together with information related to the fieldof view of the image sensor used to capture the image data. In yetanother example, the time of the activity and/or behavior may bedetermined based on the time the activity and/or behavior appears in theimage data.

In some embodiments, information received by Step 1110 or theinformation obtained by analyzing the image data received by Step 1110may comprise information about activities and/or behaviors appearing inimage data, together with identifying information of people involved inthe activities and/or behaviors. In such cases, Step 1120 may analyzethe information to identify activities and/or behaviors performed by orinvolving a selected individual or a group of selected individuals, anddetermine which of the identified activities and/or behaviors of theselected individual or a group of selected individuals repeat. In suchcases, Step 1120 may analyze the information to identify whichactivities and/or behaviors repeat, determine which activity and/orbehavior performed by which person (for example by accessing thisproperty in the data received by Step 1110 or the result of the analysisof the image data received by Step 1110), and out of the identifiedactivities and/or behaviors select activities and/or behaviors repeatedby a selected person or a selected group of people.

In some embodiments, Step 1120 may count the number of repetitions ofthe activity and/or behavior by the selected person or selected group ofpeople, and in some cases some of the activities and/or behaviors may beignored based on the number of repetitions (for example when the numberof repetitions is below a selected threshold, is above a selectedthreshold, is not within a selected range, and so forth). In someexamples, the information received by Step 1110 or obtained by analyzingthe images obtained by Step 1110 may comprise properties of occurrencesof the activity and/or behavior, and Step 1120 may count repetitionswith selected properties, for example repetitions within selected timeframes, at selected locations, and so forth. In some examples, theinformation received by Step 1110 or obtained by analyzing the imagesobtained by Step 1110 may comprise properties of the occurrences ofactivities and/or behaviors, and Step 1120 may ignore occurrences ofactivities and/or behaviors with selected properties, for example Step1120 may ignore occurrences of the activity and/or behavior that arewithin or outside some selected time frames, that are within or outsidea selected area, that involve selected objects, that occur for less thana selected minimal time duration, and so forth.

In some embodiments, determining properties of the repeated activity(Step 1130) may comprise determining properties of the repeated activityand/or behavior identified in Step 1120, for example based on theinstances of the repeated activity and/or behavior identified in Step1120 and/or based on properties of the instances of the repeatedactivity and/or behavior identified in Step 1120 and/or based on theimage data received in Step 1110. Step 1130 may be repeated fordifferent repeated activities and/or behaviors identified by Step 1120,for the same repeated activity and/or behavior of different people ordifferent groups of people, and so forth. The different results obtainedfrom the multiple repetitions of Step 1130 may be compared.

In some examples, Step 1130 may analyze the occurrence time of theinstances of the repeated activity and/or behavior, or the number ofinstances of the repeated activity and/or behavior identified within aselected time frame, to determine a frequency of the repeated activityand/or behavior. For example, Step 1130 may determine a typical oraverage frequency of the repeated activity and/or behavior for differenttime frames, for different parts of the day, for different days of theweek or month, for different months, for different seasons of the year,for different years, and so forth. In another example, Step 1130 maydetermine typical times when the repeated activity and/or behaviorusually take place, typical times when the repeated activity and/orbehavior usually take place at a selected location and/or area, and soforth.

In some examples, Step 1130 may analyze the locations that Step 1120determined for the instances of the repeated activity and/or behavior,or the information received by Step 1110 may be accessed to determinelocations for the instances of the repeated activity and/or behavior, todetermine information related to locations and areas related to therepeated activity and/or behavior. For example, Step 1130 may determinetypical location or area where the repeated activity and/or behaviorusually take place, typical location or area where the repeated activityand/or behavior usually take place at selected time frames, and soforth.

In some examples, Step 1130 may analyze properties of the instances ofthe repeated activity and/or behavior, whether determined by Step 1120or obtained by analyzing the information received by Step 1110, todetermine typical and/or aggregated and/or statistical informationrelated to the repeated activity and/or behavior. For example, suchproperties may include type of objects used in the instances of therepeated activity and/or behavior, and the determined information mayinclude a list of typical objects used, number of objects used, numberof times selected objects were used, percentage of the instances of therepeated activity and/or behavior where selected objects were used, andso forth. In another example, the repeated activity and/or behavior mayinvolve interaction with other people, and the properties may includeidentifying information about people that were interacted with, and theinformation may include a list of people that were interacted with,number of people that were interacted with, percentage of the instancesof the repeated activity and/or behavior where a selected person orselected group of people were interacted with, and so forth.

In some embodiments, providing information based on the determinedproperties (Step 1140) may comprise providing information related torepeated activities and/or behaviors identified by Step 1120 (forexample information determined by Step 1130), to a user, to anotherprocess, to an external device, and so forth. In some examples, Step1140 may provide the information visually, for example using a graphicaluser interface, using a web site, using a display system, using anaugmented reality system, using a virtual reality system, in a printedform, and so forth. For example, Step 1140 may visually present imagesof the repeated activities and/or behaviors (for example, parts ofimages received by Step 1110 that depicts the repeated activities and/orbehaviors). In another example, Step 1140 may visually present textualinformation describing the repeated activities and/or behaviorsidentified by Step 1120 and/or properties of the repeated activitiesand/or behaviors determined by Step 1130. In yet another example, Step1140 may present a graph comparing properties of repeated activitiesand/or behaviors determined by Step 1130. In some examples, Step 1140may provide the information audibly, for example through audio speakers,using head set, and so forth. For example, textual informationdescribing the repeated activities and/or behaviors identified by Step1120 and/or properties of the repeated activities and/or behaviorsdetermined by Step 1130 may be read aloud, for example by taking thetextual information and converting it to audible output using text tospeech algorithms.

In some embodiments, Step 1120 and/or Step 1130 may be repeated toidentify different repeated activities and/or behaviors of the sameperson or of different people, or to identify the same repeated activityand/or behavior of different people. Process 1100 may receive (from auser, from another process, from external device, etc.) a request toprovide information related to a selected person and/or a selectedactivity, and Step 1140 may provide the requested information. Forexample, a first request for information related to a first person maybe received, and in response to the first request Step 1140 may provideinformation related to properties of a repeated activity of the firstperson, and a second request for information related to a second personmay be received, and in response Step 1140 may provide informationrelated to properties of a repeated activity of the second person. Inanother example, a first request for information related to a firstactivity and/or behavior may be received, and in response to the firstrequest Step 1140 may provide information related to properties of thefirst activity and/or behavior, and a second request for informationrelated to a second activity and/or behavior may be received, and inresponse Step 1140 may provide information related to properties of thesecond activity and/or behavior. In yet another example, a first requestfor information related to a first person and a first activity and/orbehavior may be received, and in response to the first request Step 1140may provide information related to properties of the first activityand/or behavior of the first person, and a second request forinformation related to a second person and a second activity and/orbehavior may be received, and in response Step 1140 may provideinformation related to properties of the second activity and/or behaviorof the second person.

FIG. 12A is a schematic illustration of an example image 1200 capturedby an apparatus, such as apparatus 200. In this example, image 1200 maycomprise person 1202 and person 1204 watching television. FIG. 12B is aschematic illustration of an example image 1210 captured by anapparatus, such as apparatus 200. In this example, image 1210 maycomprise person 1212 and person 1214 engaging in a physical activity, inthis case playing soccer. FIG. 12C is a schematic illustration of anexample image 1220 captured by an apparatus, such as apparatus 200. Inthis example, image 1220 may comprise person 1222 and person 1224interacting with each other, in this case in a conversation. FIG. 12D isa schematic illustration of an example image 1230 captured by anapparatus, such as apparatus 200. In this example, image 1230 maycomprise person 1232 and person 1234 sitting next to a table and eating.FIG. 12E is a schematic illustration of an example image 1240 capturedby an apparatus, such as apparatus 200. In this example, image 1240 maycomprise person 1234 sitting next to a table and eating. FIG. 12F is aschematic illustration of an example image 1250 captured by anapparatus, such as apparatus 200. In this example, image 1250 maycomprise person 1232 sitting next to a table and eating. Process 1100may obtain images 1200 and/or 1210 and/or 1220 and/or 1230 and/or 1240and/or 1250 using Step 1110.

In some examples, Step 1120 may analyze image 1200 to identify thatperson 1202 and person 1204 are watching television, and may furtheridentify properties of this activity, such as the time the activitybegins and/or ends, the duration of the activity, the identity of thepeople watching the television, the identity of people present in theroom that do not watch the television, the content watched in thetelevision (for example by analyzing images of the television screen andcomparing it with a database of known contents to identify the content,by analyzing audio and comparing it with a database of known contents toidentify the content, by receiving content identifier from thetelevision and/or from a device paired with the television, etc.), thesitting arrangement, the food consumed while watching the television,and so forth. Other images obtained by Step 1110 of people watching thetelevision may be analyzed in a similar manner, including images werethe people watching the television include person 1202 and images werethe people watching the television do not include person 1202. Step 1130may aggregate information about all or some of the occurrences wereperson 1202 watched the television. For example, Step 1130 may generateaggregated information and/or statistics about the watching habits ofperson 1202, such as the watching hours, the watching durations,watching mates, watched content, sitting place, food consumed duringwatching (for example, the average calories per day consumed whilewatching television), and so forth. The information generated by Step1130 may be provided using Step 1140.

In some examples, Step 1120 may analyze image 1210 to identify thatperson 1212 and person 1214 are engaged in a physical activity, and mayfurther identify properties of the physical activity, such as the timethe activity begins and/or ends, the duration of the activity, theidentity of the people engaged in the physical activity, the identity ofpeople present in the environment that are not part of the physicalactivity, the type of the physical activity (playing soccer in theexample of image 1210), the location the physical activity takes placeat, and so forth. Other images obtained by Step 1110 of people engagedin physical activity may be analyzed in a similar manner, includingimages were the people engaged in the physical activity includes person1212 and images were the people engaged in the physical activity do notinclude person 1212. Step 1130 may aggregate information about all orsome of the occurrences were person 1212 was engaged in physicalactivity. For example, Step 1130 may generate aggregated informationand/or statistics about the exercising customs of person 1212, such asexercising hours, physical activities durations, exercising partners,exercising locations, tools used for exercising, type of physicalactivities, and so forth. The information generated by Step 1130 may beprovided using Step 1140.

In some examples, Step 1120 may analyze image 1220 to identify thatperson 1222 and person 1224 are interacting with each other (in thisexample in a conversation) , and may further identify properties of theinteraction, such as the time the interaction begins and/or ends, theduration of the interaction, the identity of the people involved in theinteraction, the identity of people present in the room that are notinvolved in the interaction, the type of the interaction (such asconversation, hand shake, etc.), the content of a conversation (forexample by analyzing audio captured from the environment using speech totext algorithms and/or natural language processing algorithms), thelocation of the interaction, and so forth. Other images obtained by Step1110 of people interacting with each other may be analyzed in a similarmanner, including images were the interacting people include person 1222and images were the interacting people do not include person 1222. Step1130 may aggregate information about all or some of the interactionsinvolving person 1222. For example, Step 1130 may generate aggregatedinformation and/or statistics about the interactions of person 1222,such as the times of the interactions, the durations of theinteractions, the partners to the interactions, the content of theconversations, the locations of the interactions, and so forth.

In some examples, Step 1120 may analyze image 1230 to identify thatperson 1232 and person 1234 are eating, and may further identifyproperties of the meal, such as the starting and/or ending time of themeal, the duration of the meal, the identity of the people eating, theidentity of people present that do not eat, the food consumed by aperson and/or properties of the food consumed (such as the calories ofthe food, nutrition value of the food, the ingredients of the food,etc.), the sitting arrangement, and so forth. Other images obtained byStep 1110 of people eating may be analyzed in a similar manner. Forexample, Step 1120 may analyze image 1240 to identify that person 1234is eating while person 1232 is not present, and may analyze image 1250to identify that person 1232 is eating while person 1234 is not present.Step 1130 may aggregate information about all or some of the occurrenceswere person 1232 is eating. In such case, information from images 1230and 1250 may be taken into account and information from image 1240 maybe ignored, while when Step 1130 aggregates information about all orsome of the occurrences were person 1234 is eating, information fromimages 1230 and 1240 may be taken into account and information fromimage 1250 may be ignored. For example, Step 1130 may generateaggregated information and/or statistics about the eating habits ofperson 1232, such as meals time, meals duration, food consumed (orproperties of the consumed food, such as calories, nutrition values,ingredients, etc.), sitting place, and so forth. The informationgenerated by Step 1130 may be provided using Step 1140.

In some examples, Step 1120 may analyze image 1030 to identify thatperson 1032 is using computerized device 1034, and may further identifyproperties of this activity, such as the time the activity begins and/orends, the duration of the activity, the type and/or identity ofcomputerized device 1034, usage data of computerized device 1034 (forexample by analyzing image of the computerized device 1034 screen andcomparing it with a database of known applications, by receiving usageinformation from computerized device 1034, etc.), the location of theactivity, and so forth. Other images obtained by Step 1110 of people(whether person 1032 or other persons) using computerized devices(whether computerized device 1034 or other computerized devices) may beanalyzed in a similar manner. Step 1130 may aggregate information aboutall or some of the occurrences were person 1032 used computerized device1034 or of all or some of the occurrences were person 1032 used anycomputerized device. For example, Step 1130 may generate aggregatedinformation and/or statistics about the computerized device usage habitsof person 1032 or about usage habits of computerized device 1034 ofperson 1032, such as the usage hours, the usage durations, the usedcontent, the usage locations, the used devices, and so forth. Theinformation generated by Step 1130 may be provided using Step 1140.

It will also be understood that the system according to the inventionmay be a suitably programmed computer, the computer including at least aprocessing unit and a memory unit. For example, the computer program canbe loaded onto the memory unit and can be executed by the processingunit. Likewise, the invention contemplates a computer program beingreadable by a computer for executing the method of the invention. Theinvention further contemplates a machine-readable memory tangiblyembodying a program of instructions executable by the machine forexecuting the method of the invention. Moreover, consistent with otherdisclosed embodiments, non-transitory computer readable storage mediamay store program instructions, which may be executed by at least oneprocessor and perform any of the methods described herein.

What is claimed is:
 1. A system for image processing, the systemcomprising: at least one processing unit configured to: obtain a firstgroup of one or more images captured using at least one image sensorfrom an environment; analyze the first group of one or more images toobtain scene information; based on the scene information, obtain atleast one inference model; use the scene information to determineparameters for modifying the obtained at least one inference model;obtain a second group of one or more images captured using the at leastone image sensor from the environment; and process the second group ofone or more images using the at least one inference model.
 2. The systemof claim 1, wherein the system includes the at least one image sensor.3. The system of claim 1, wherein the second group of one or more imagesis captured after the scene information is obtained.
 4. The system ofclaim 1, wherein obtaining the at least one inference model comprisesselecting the at least one inference model of a plurality of alternativeinference models based on the scene information.
 5. The system of claim1, wherein obtaining the at least one inference model comprises:transmitting to an external apparatus the scene information; andreceiving at least part of the at least one inference model, where thereceived at least part of the at least one inference model is based onthe transmitted scene information.
 6. The system of claim 1, whereinobtaining the at least one inference model comprises: obtaining trainingexamples based on the scene information; and generating at least part ofthe at least one inference model using the training examples.
 7. Thesystem of claim 1, wherein the scene information comprises at least anindication of a swimming pool, and wherein processing the second groupof one or more images using the at least one inference model isconfigured to detect drowning events.
 8. The system of claim 1, whereinthe scene information comprises at least an indication of a piece ofequipment, and wherein processing the second group of one or more imagesusing the at least one inference model is configured to detect safetyrelated events associated with the piece of equipment.
 9. The system ofclaim 1, wherein the scene information comprises at least an indicationof at least one of a pet and a child.
 10. The system of claim 1, whereinthe scene information comprises at least one of a property of an objectappearing in the first group of one or more images and a property of aperson appearing in the first group of one or more images.
 11. Anon-transitory computer readable medium storing data and computerimplementable instructions for carrying out a method, the methodcomprising: obtaining a first group of one or more images captured usingat least one image sensor from an environment; analyzing the first groupof one or more images to obtain scene information; based on the sceneinformation, obtaining at least one inference model; using the sceneinformation to determine parameters for modifying the obtained at leastone inference model; obtaining a second group of one or more imagescaptured using the at least one image sensor from the environment; andprocessing the second group of one or more images using the at least oneinference model.
 12. The non-transitory computer readable medium ofclaim 11, wherein the second group of one or more images is capturedafter the scene information is obtained.
 13. The non-transitory computerreadable medium of claim 11, wherein obtaining the at least oneinference model comprises selecting the at least one inference model ofa plurality of alternative inference models based on the sceneinformation.
 14. The non-transitory computer readable medium of claim11, wherein obtaining the at least one inference model comprises:transmitting to an external apparatus the scene information; andreceiving at least part of the at least one inference model, where thereceived at least part of the at least one inference model is based onthe transmitted scene information.
 15. The non-transitory computerreadable medium of claim 11, wherein obtaining the at least oneinference model comprises: obtaining training examples based on thescene information; and generating at least part of the at least oneinference model using the training examples.
 16. The non-transitorycomputer readable medium of claim 11, wherein the scene informationcomprises an indication of a swimming pool, and wherein processing thesecond group of one or more images using the at least one inferencemodel is configured to detect drowning events.
 17. The non-transitorycomputer readable medium of claim 11, wherein the scene informationcomprises an indication of a piece of equipment, and wherein processingthe second group of one or more images using the at least one inferencemodel is configured to detect safety related events associated with thepiece of equipment.
 18. The non-transitory computer readable medium ofclaim 11, wherein the scene information comprises an indication of atleast one of a pet and a child.
 19. The non-transitory computer readablemedium of claim 11, wherein the scene information comprises at least oneof a property of an object appearing in the first group of one or moreimages and a property of a person appearing in the first group of one ormore images.
 20. A method comprising: obtaining a first group of one ormore images captured using at least one image sensor from anenvironment; analyzing the first group of one or more images to obtainscene information; based on the scene information, obtaining at leastone inference model; using the scene information to determine parametersfor modifying the obtained at least one inference model; obtaining asecond group of one or more images captured using the at least one imagesensor from the environment; and processing the second group of one ormore images using the at least one inference model.